Templatize UI

[cecilia-donnelly]

We've done some UI templating already, in #173 and #202. Ultimately, we want to understand the current UI-generation system and possibly update it. So, this issue is in two parts:

1. How is the UI is being generated now? What needs to be done to fully templatize it? The short answer to this, from @jasonaowen, is "via JSPs and CSS."

2. Are there "cross-platform templating solutions"? i.e., can we use a template that can be generated by Java and Python code? For example, if we ever want to put a web interface on the ETL process, which is being done with Python, it should use the same templates that we use for the main app, even though that's written in Java.

[jasonaowen]

Apache Velocity is a templating engine that we depend on. The last release was in 2010. Our use of it is limited to formatting the emails we send: new registration and password changes. We're not using it for rendering any web pages, and since it seems to be an abandoned project, we probably should not start doing so.

[jasonaowen]

How is the UI being generated now?

The HTML rendered by the PSM application uses the model-view-controller pattern, using Spring for the model and controller, and JSPs for the view. There are several thousand lines of CSS to describe the HTML elements in the JSPs.

What needs to be done to fully templatize it?

Let's step back a moment and define our goals. The problem I have heard described before is:

The PSM will be deployed in different states. Each state will have its own identity standards, or style guide, or color palette, or similar, which the PSM should follow so as to look like part of a cohesive whole, rather than a disparate, disconnected module.

I suggest that it would be useful to get copies of those identity guidelines sooner rather than later, as the depth they go into and the variety between them will inform our solution.

Without having seen any such state brand guidelines (despite a few minutes' searching), I would guess that there are a few key points:

• logo
• fonts
• colors

All of these will need to be data-driven rather than code-driven (PR #173 demonstrates how the logo is currently code-driven).

The logo is probably the most obvious and most important. It will be an image file, which we currently use CSS to set to a fixed size; that size will likely need to become data-driven. We might be able to use an image hosted outside of the application, and that will likely be both simpler and more performant; that option will depend on a conversation with states' IT departments. If so, we'll need to allow configuration of the logo URL; easiest is a build property, next would be an environment variable of some sort, and hardest would be to build out UI for it. If we need to host the logo file ourselves, we can bundle it in at build time, but if it needs to be dynamic at runtime we'll need to build an endpoint to allow downloading it, as well as some way to upload it - a script, an API, or UI.

The colors are the second most important, and likely the most difficult. We will need to make several CSS rules dynamic rather than static, but figuring out which may be challenging: there are hundreds of JSP fragments and thousands of lines of CSS. Again, having a properties file to edit which is compiled to CSS at build time is likely the simplest; however, if that does not allow for a quick enough turnaround time for designers / administrators, we could instead build a UI around setting these properties.

The fonts will also need to be hosted somewhere; like the logo, that might be inside or outside the app. Depending on the depth of the guidelines around fonts, we might need multiple fonts or only a single font, and we'll need to change more or less CSS to suit. In my mind, fonts are the least important of the potential identity guidelines.

Cross-platform templating solutions

There are a few options here, again depending on what exactly our goal is.

We could reuse the logo, CSS, and fonts, simply by referencing the PSM's resources from other applications. This is likely the simplest, and would require the least change to the PSM app itself.

As a variant on that, we could extract an application for state-specific styles, which may address some of the issues around building UI for administrators to select these styles: while the PSM is deployed as an app in /cms, we could build and deploy another app in /style or similar which allows changing these settings and allows any application on the same domain to read them.

If we want deeper integration, we can use the existing JSPs from Python with Jython. JSPs have been around for a long time, as has Jython, so support for this should be mature. After briefly reviewing our JSPs, though, I'm not convinced that they are worth the effort of reusing them, as they are tailored to the application rather than to reuse - we would need to refactor them to be useful.

In the other direction, we can use Jinja2 in Java, altering the application to use Jinja2 templates rather than JSPs. I would hope we could do that only for the parts we want to reuse, and leave the rest as JSPs, since those are working as-is.

[cecilia-donnelly]

As a variant on that, we could extract an application for state-specific styles, which may address some of the issues around building UI for administrators to select these styles: while the PSM is deployed as an app in /cms, we could build and deploy another app in /style or similar which allows changing these settings and allows any application on the same domain to read them.

I like this idea. Any other module could then rely on this dynamic-content-loading app, and admins could upload a logo and change colors/fonts there.

@slifty, @jvasile, what do you think?

[jasonaowen]

The riskiest changes, in my mind, are the CSS rules around color. I'll look into switching some of those around tomorrow to see how hard it is to change them site-wide.

[jasonaowen]

In the hope that this is interesting to other people...

There are around 40 distinct colors spelled out by hex code in our CSS:

$ grep -o '#[0-9a-fA-F]\{3,6\}[ ;]' style.css \
    | sed -e 's/#//; s/ //; s/;//;' \
    | tr '[:upper:]' '[:lower:]' \
    | sort \
    | uniq -c 
      6 000
      1 000001
      9 064265
      5 0d4478
      5 1aabff
      2 51650b
      1 5bc217
      1 666
      2 666666
      1 698800
     29 7ea5c8
      1 939393
      2 999
      7 999999
      2 acb785
      2 bcb870
      6 bdbdbd
      2 c00
      1 cccccc
     84 d2d2d2
      1 d5e2e5
      2 da1820
      1 dbdbdb
      4 ddd
      1 dddddd
      1 dedcdc
      1 ececec
      2 ecf2f3
      2 f00
      1 f6f6f6
      1 f7f7f7
      1 f9f9f9
      1 faf9f9
      2 fcfbea
      1 ff1010
      2 ff2f2f
      1 ffb0bb
     17 fff
      1 fffffd
      5 fffffe
      9 ffffff

[jasonaowen]

One example of a style guide is the HHS.gov Style Guide. It covers the above topics (logo, colors, and fonts), as well as several other layout and style considerations. (Thanks to @cecilia-donnelly for the link!)

[jasonaowen]

After reviewing the CSS that we have, I feel comfortable saying I would be able to modify it to change colors to match a style guide's color palette -- however, it would probably take me a long time. CSS is not one of my stronger areas, and while I'm sure I could make it work, I suspect others would be able to do so in a more cost-effective manner, both for this change and to enable future changes. If possible, I suggest finding a front end developer who is more proficient in CSS.

[jasonaowen]

I discovered another alternative today: WildFly deployment overlays allow replacing specific files at deploy time without modifying any source code or artifacts. Once we've extracted the replaceable colors into their own CSS file, states could copy it out of the repo, modify it as needed, and use it in a deployment overlay.

[jvasile]

Jason and I talked about this issue, and this note is just to get it on the record for others to see.

1. We should template-ize the css. That's a relatively small lift and can be done by regenerating the style.css file that gets served as a static item by the Wildfly server. We can store config data in a new table in the db.

2. Template-izing the html is harder. The pages are not well factored. One approach is to generate one page from a template and then refactor as you add a second, then a third, etc. The goal is to be able to easily add pages and to have all the pages get generated the same way instead of the somewhat ad hoc way how they are now. We're going to leave that step until after (1).

Doing this isn't on Jason's plate, at least for now. He and I agreed he's not the natural choice and so he's going to keep plugging on his other work. Despite that, his knowledge of the codebase and db were super valuable in thinking this stuff through. He remains available for further consultation.

[brainwane]

Our choice of templating languages has implications for our choices of user documentation tooling (as discussed in #195 and #265). For now I'll use reStructuredText as I write (it'll be easy to convert from .rst to any other file format using pandoc, and Sphinx natively reads .rst and Read The Docs integrates well with Sphinx), but I'll keep an eye on this issue to see if I should rethink this.