Forms-heavy application with auto-save

[sousekd]

I am developing a forms-heavy application on Aurelia, which I am happy user of since its first non-beta release. The similar applications I’ve developed so far were pretty simple from architectural perspective, with a lot of forms and dialogs, all of which typically had the Save and possibly Cancel button to submit or reset changes. For canceling I have either kept the copy of original form data in app, or I have just reloaded the data from back-end. Irrespective of that, form controls were bound to view model, and the Save buttons transformed data from view model to a shape required by back-end API (typically one call per form / Save button press).

This time, I would like to do things a bit differently and achieve the “auto-save” functionality for most forms. Although the data model I need to work with is quite complex with many structures, arrays and cross-dependencies, the workflow is simple and allows me to avoid any client side validation. Because of the model size, I wouldn’t be able to afford auto-saving on the top-level entity, so I will need to have and call APIs for partial modifications.

I don’t have any experience with Aurelia Store (or the concept in general), but if it can help, it would be a good opportunity to learn something new. Otherwise, I was thinking about developing some sort of recursive “Deep Observer” using Aurelia Binding Engine, which would be able to observe changes on complex objects. I would either instantiate such observers on the higher-level screens to detect changes and perform Save operations as necessary, or maybe let some “service components” do the same to keep forms free of any Save related code…

Or is it completely wrong idea? Did you faced the similar challenge? Can you point me in the right direction?

[CuddleBunny]

I think you’re headed in the right direction. I would personally recommend using Aurelia Store or similar in a complex form wizard just because it helps separate concerns. I work for an insurance company with many of these sorts of applications and most of the contention comes from making global changes to pages with baked in functionality.

Some notes from your thoughts and approaches that I’ve found useful:

• One big model on the client and server. Client uses TypeScript’s Partial<T> and server (.net) uses view models.
• Server model is significantly larger. It contains rules, computed properties, among other things. These are all superfluous to the UI and should not be sent down to the front end.
• Don’t write off client-side validation. There are many scenarios where validation is more for UX than anything so I do those on the client.
• If you have branching logic in your forms where choices affect which subsequent pages are displayed you should represent this with a decision tree or state machine of some sort rather than using conditional logic on each page.
• If you have non-blocking API calls/deferred validations run those async in the background and keep a list of promises so you can stop if the user hits the point where they should be done too fast.
• For save/resume type scenarios you can either track change events and replay them to rebuild your state or figure out if you can grok the app state from the model state. (Like if property X is defined, user must be on page Y)

[jeremyholt]

I would definitely suggest using aurelia-store.

Make sure you always get a deepcopy of the state in your viewmodel and only bind to that. (lodash _.cloneDeep). All the literature says do const newState = {...state} but it causes a false sense of security because it is not cloning deep models.

The form’s bind() is where you clone your model from the state. So you can either call your backend in activate() or in bind() itself. bind() is the first place you are guaranteed that state is not undefined, if you are using @connectTo. If you manually subscribe to store.state you can do it in the constructor, and have the state available in activate().

If you make state observable, be aware that it will fire anytime something happens to the state, which is a nightmare to debug.

One of the things I’ve found using aurelia-store, is that I have dramatically reduced my need to route with parameters - most times I can deduce the parameters for my call to the backend from the state itself since I should know where the route originated from.

The form is saved either by clicking the save button or on deactivate.

Prior to saving I use a combination of client-side validation, and a simple isDirty check using lodash _.isEqual, to ensure that I’m not saving the data twice.

The back end does all the heavy validation and sends exceptions (of which there are very few) back to the client which are then converted into actions that update the store and inform the client.

It took me long long time to get my head around aurelia-store. The dox are very good, but don’t do a very good job in explaining how and why you should consider using store. In my view there should be a clear explanation at the start of the dox, but instead it gets into all the esoteric details of configuration from the first paragraph on.

[zewa666]

Thanks for the feedback on the docs. Tbh I’ve spent double the time writing the docs vs implementing the store plugin so I’d be very curious which exact parts you felt were overloaded or not explaining enough of the background. You’ve explicitly mentioned the reasons to use state management as described in the intro. Would you mind elaborating what is missing for you? Perhaps even better to keep the details together would be if you could open up a new issue in the store repo so we get to the details over there. Thx in advance

[jeremyholt]

Hey - I didn’t mean to criticize the documents, and made my comments above from memory! Having gone back and looked at the opening paragraphs I think my criticism was unjust. Have the dox changed recently?

To be honest, when I go to the dox page now, I just scroll down looking for some configuration detail.

However, I do have a big question which I allude to in my comments.

Using connectTo() only sets up state from bind() onwards. From experience there seems to be no reason why the state cannot be initialized in the ctor, other than the fact that you have to manually setup and teardown the subscription in unbind() or detached().

Would it not be possible to have connectTo() do it’s magic in the ctor instead of bind()?

Also, is there no mechanism to auto register the actions? It’s a lot of extra typing to store.registerAction("action",action)

With state setup in the ctor(), activate() now has access to it. One of the things I’ve tried to do is reduce as much as possible passing parameters through routes to views, much preferring (where possible) to pick up the “params” from the state. Personally I find managing the parameters for routes a bit of a nightmare, because I cannot use a rest backend. I use RavenDb, whose Id is of the format customer/1-A, which means that I have to send the query to the backend with the parameters encoded as query variables. Not a big deal - I just need to remember to do it!

The other “issue” I have is accessing state within the service. It would be nice if there was some shortcut to doing this. At the moment I set up a subscription for the state, extract the readonly property I’m interested in (effectively a global variable), and then tear down the subscription - all in the same method call.

My biggest complaint about aurelia-store is the same I would make against Aurelia in general - most of the time it so flexible, there are so many different ways of getting to the same final result (note I did not say there are so many different ways to do the same thing), that it can be terribly confusing for a newbie. I’ve been using Aurelia since it first came out, and still most definitely consider myself a newbie. I learn new stuff all the time from comments in the these threads - particularly you and @bigopon and @khuongduybui and @huochunpeng comments.

The biggest initial confusion I had was what I should be doing in the action. I had always used a service to call the back end, and returned the data to the viewModel directly from the service. This of course led to the problem that aurelia-store resolves so well.

At one point, I must have read the dox to mean that I should make the calls to the backend in the action, and spent a long time struggling with this, since it was so contrary to what I had been doing until then. I then read a comment by @khuongduybui in one of the threads that completely opened my eyes - it allowed me to continue to use my services as I was comfortable with - viewModel asks service for data => service asks back end for data : service “massages” the data from the server => service sends action updating the state.

My services are now basically one liners, and the actions do nothing more that clone the state, update the state with the new data, and return the cloned state.

To close, I guess it would be nice for the dox to have some broad overview of the possible “best practices” at the start, before delving into rxjs observables (which I still don’t understand - and frankly, don’t have to understand to use the store).

I would also warn in capital letters NOT TO BIND.TWO-WAY DIRECTLY TO STATE - the problem is you don’t necessarily realize that you are doing it until it’s too late, particularly if you haven’t cloned the state properly {… state}.

As a belts and braces measure, in the viewModel I clone the state that I’ve already cloned in the action again, just to make double sure I’m binding to a truly different object.

Apologies for the ramble …

Jeremy

[zewa666]

No offsense taken, but lets perhaps move this conversation into a new topic, or if it’s still more about documentation as suggested into a github issue. I’d rather not spam this topic from @sousekd with this sidetracks.

[sousekd]

Ok, @CuddleBunny, @jeremyholt, thank you for your input.

I have read through the documentation of Aurelia Store, but I am still not sure whether it is intended or suitable to help me with what I am trying to achieve. I guess every app is different, but my point of view is limited to the kind of applications I develop, all the time. I guess one picture may be better than a thousand words, so here is the screenshot of the application I am developing. I have changed the entity names to something more generic to avoid issues with NDA:

This is the typical screen showing detail of one of the top-level entities, Country. There are couple of tabs for Country details, with the Cities tab selected. The list on the left shows the list of the Cities, filterable. Here users are able to delete, create and reorder Cities, and they can select one. Once they do so, the right side of the screen shows the City detail. Again, there are City related tabs, with the Street tab active. List of Streets, and below the detail of selected Street, with couple of tabs.

Some tabs will have lists of child entities, some will be “forms”, which can vary in complexity. Depending on the user rights and entity state, some forms will be read-only, some editable. When a form is edited, it should auto-save without the need to press the (non-existing) Save/Submit button. When adding a new record, the application would either add a new row to the corresponding list (and select it), filling-in default values, or it would display a modal dialog with the Save/Submit button, for entities where some validation needs to occur at the time of creation, and cannot be delayed.

I need to provide deep-linking, so I use parametrized child-routers on every “index” screen with tabs (here for Country, City and Street), and on every “split” screen with list and detail of selected item (here for Cities and Streets). I am trying to design the app in a way it allows opening forms in a different container. If another top-level entity would have a list of Street references, I would like to be able to use the existing Street “index” component on that different screen (just an example).

Traditionally, I would probably inject a sort of CountryService into the Country “index” screen, which would use it to load the “current” Country using route parameters in Activate. The same CountryService would be injected into the Cities “split” screen, which would also inject CityService and use them both to get the list of the Cities in the current Country. Selection of a City would cause a router to navigate to the City “index” screen, which would load the “current” City using the injected CityService and the route parameter. And the same would apply lower in the hierarchy. I may be simplifying, and *Service may not be the correct naming, but you got the idea…

One challenge I see here is that I am not sure how I would propagate data changes on entities lower in the hierarchy to visible elements higher in the hierarchy. For example, if I change Street name using the form, it should immediately change in the list of Streets. But the list will probably use a different source of data. I would likely end-up with reloading the list using Signaler to avoid the mess of combining list data with the current Street detail view-model. The same may apply for various status information, aggregated numbers, results of server side validation etc.

The other challenge obviously is how to detect changes to a form and auto-save it. As noted, I plan to write a “simple” deep-observer using Aurelia BindingEngine which would detect changes on all object properties incl. arrays recursively, with the option to exclude some paths from observation. On the “right” places, I would use it to observe view-model and execute Save API. The “right” place may differ from entity to entity (or screen to screen), depending on complexity and workflow.

All that said, I would be interested to hear architecture ideas from more experienced devs here.
And I would love to hear, whether Aurelia Store is the right fit for this application, and how it could make my life easier :-).

Thank you so much for help!

[zewa666]

Have you built that table by yourself or are you using a grid plugin? @ghiscoding can perhaps tell more about editable columns and potential use cases for a onChange handler

[bigopon]

You don’t need to write your own deep observation, we have a few built-in stuff for this. I think I can guess what you want to achieve vaguely, but it would be a lot easier if you can provide some pseudo code via a sandbox (fork & edit), either:

• [TypeScript] Sandbox
• [JavaScript] Sandbox

It doesn’t need to work, just enough to demonstrate where the crux is

[jeremyholt]

It looks horribly like the sort of applications I write!!

As a start I would suggest that you create your state to look something like

export interface IState {
country:{
       current: ICountry;
       list: ICountryListItems[];
     };
city: {
      current: ICity;
      list: ICityListItems[];
  };
}

initialState

const initialState: IState = {
   country:{
      current: undefined,
      list: []
  },
  city: {
      current: undefined,
       list:[]
  }
}

Your countryService would get the country from the backend and send an action to update the country part of the state. It would also be responsible for updating the list of countries.
Same thing for the citiesService.

Your viewModel would bind to the clones of state.country.current and state.city.current.

protected bind(){
   this.model=_.cloneDeep(this.state.country.current);
}

When you make a change to your model and send it back to your service, it updates the state, and your viewmodel will update accordingly

@connectTo()
export class MyViewModel {
   @observable state: IState;
   public modeCountryl: ICountry;
    public modelCity: ICity

  protected stateChanged(state: IState){
   /// do something here - bear in mind that this will be hit every time state anywhere is changed.
      this.modelCountry =_.cloneDeep(this.state.country.current);
      this modelCity = _.cloneDeep(this.state.city.current);
  }

  protected bind(){
    this.modelCountry = _.cloneDeep(this.state.country.current);
    this modelCity = _.cloneDeep(this.state.city.current);    
  }

  protected selectCountry(item: ICountryListItem){
      this.myCountryService.getCitiesFor(item.cityId);

      // this is where aurelia-store really shines - your state is updated in stateChanged(state: IState), and your model is updated, and so are your bindings.
      // usual caveats for binding to arrays - you can use the bindingEngine for that, or splice().
   }

protected saveCountry(){
   // which will send an action in your service to update state.country.current
   this.countryService.save(this.modelCountry)
  }
}

Since you will always have something for current, both for your state.country and state.city, you can probably get away with not needing to send parameters with the routes, so instead of

protected activate(params: IParams){ 
    myService.getCity(params.cityId)
}

you can do
if you are using connectTo() you need to do this in bind() because that is the first time state is defined

protected bind(){
   myService.getCity(this.state.city.current.id);
}

OR if you want to this in activate (and have access to state in canActivate as well) you need to manually set up the subscription to state in the ctor

constructor(
private store: Store<IState>
){
  this.subscriptions.push(store.state.subscribe(state=>this.state=state);
}

protected unbind(){
  this.subscriptions.forEach(c=>c.unsubscribe());
}

From my own experience, it seems far more complicated than it really is. It took me about 2 months of slogging away at it before I had my “AHA” moment.

I think the above pseudo code allows you to do what you want, namely update different parts of your model independently of each other, using the state as the “arbiter of truth”.

[ghiscoding]

I’m not sure that my grid (Aurelia-Slickgrid) would help in this case, because my lib is a wrapper of SlickGrid which is a jQuery lib and the editors in my case are all written in plain JS/jQuery, then SlickGrid has some events (onCellChange and others) that we can subscribe to and from there I call the backend and usually refresh that 1 line (not the entire grid) that I was just editing just to make sure that I’m in synch with the backend DB.

I find there’s too much info shown in that print screen. In our project we use a single grid and when the user wants more info, he can click on the row and that will reveal a quick view (sidebar bar on the right and we just use Bootstrap container, just change the grid container from col-md-12 to col-md-8 and use col-md-4 for the quick view, once it get closed then the grid will resize to col-md-12). The quick view is meant to display more information about that row, there’s also a 2nd grid of items in that quick view. But as the name suggest it’s a quick view and if the user wants to see the full picture, he’ll click on Open Detail which will route to another page with more info… So anyway, I find this screen a little too busy and that’s probably why there’s performance issues at hand.

[sousekd]

Grids (or lists) are just plain HTML tables with some CSS hacks to keep the header row sticky, and they will not be editable. I prefer such before full-featured Grid components wherever possible because I find it difficult to style such components to my need (I hate styling), and their APIs are not very Aurelia friendly. We have some custom attributes to add functionality to plain tables like sorting on columns, ability to select row(s), virtual scrolling etc., some of which works in tandem with our small data-source component which helps with adding list-related parameters to API calls (paging and sorting related). Anyway, I don’t think we will need to use any of them in this particular app, as lists in the context of entities will only contain few items, definitely not more than 100.

That said, it will be possible to execute some quick actions directly from the lists shown here, like changing Status or Category (implemented using Bootstrap dropdowns). And we will have editable grids on the forms themselves (not shown here), but they will be developed for every form separately, Aurelia makes it easy.

[sousekd]

Yeah, a lot of things on the screen. We are rewriting an old-school Windows application which had the similar interface. It makes sense, as they work on “Country”, adding and editing “Cities”, including “Streets” (if I use entity names of the example), and they need to quickly jump from one instance to another, often copying and pasting stuff.

It might not be obvious from the screenshot, but the screen splitter can be moved, giving more space to either list or form, eventually hiding one part and providing more space for the other part. There will be buttons to quickly switch to full-screen and back. I guess we will see how well it will be received…

[sousekd]

Good to know, I will try to prepare some example. Thanks a lot!

[ghiscoding]

Yeah I did assume from your print screen that the grey bar in the middle was moveable (I thought it was because it’s wider than usual so it’s kinda of a hint I guess). I just want to come back to what I wrote a bit earlier, in our project if I open the quick view (sidebar) and do an action that could affect a row in the grid, I would simply call a method to re-render that row in the grid with the new data object of that row (internally it does a find row by ID and after finding it, it re-render that row with the new data, the grid requires unique IDs to work). Also to provide a bit more info about Aurelia-Slickgrid, it has a DataView service that has a bunch of methods I can call, that DataView holds all the data and it’s a simply javascript array, so it’s easy to find a row and update and everything else (it works great even with a million row, which is what SlickGrid advertises)… So all that to say, I don’t have, neither need, any deep binding of each row. The grid is super fast and barely has any memory usage (the only memory it takes is the javascript array that it keeps internally).

Oh and it’s also super fast because SlickGrid uses Virtual rendering which will basically render only what is visible on the screen (even if you have a million rows, it will only render on the screen about 50 rows, it will render the next bunch of rows whenever you scroll to another position). I wrote a blog post about Aurelia-Slickgrid last month.

[sousekd]

Thank you so much for the example!

I need to think about it a bit (= a lot). I need deep linking, so I need activate() to accept params and use them. But I guess it would be just fine to let selectSomething() to navigate instead of modifying the state and modify the state.currentSomething from within the activate method later (?).

I’ll need to read through the Aurelia Store doc again. I still don’t fully understand its benefits in this scenario, as I have a feeling I’d achieve the same results if I’d keep the currentSomething anywhere else, and let this “anywere else” be injected into where I need it.

I’m also puzzled by the fact it is recommended to have just one state object for the app. I kind of understand the reasoning, but do worry about 1 very long place where everything is.

[sousekd]

Yeah I have read your blog post and was happy someone did such a great work for all of us. I just don’t think it makes life easier for me in this particular scenario, where I have a simple list with no extra functionality needed. If I bind the table to an Array, and something causes the change of one item, I can just modify that item in the Array the table is bound to. The same way I would modify the DataView using your grid. Am I correct?

I’d use SlickGrid or similar if I would need more advanced features which are difficult to implement, like columns customization, grouping, inline editing with arrows navigation, stuff like that.

[ghiscoding]

@sousekd

If I bind the table to an Array, and something causes the change of one item, I can just modify that item in the Array the table is bound to. The same way I would modify the DataView using your grid. Am I correct?

Yes and No, because Aurelia-Slickgrid is wrapper of jQuery lib, there’s no direct binding to the array and most of the events are done the old fashion way with dispatch events. You understood correctly about the DataView in SlickGrid, but the main difference is that there are events which are published every time a DataView method is called (adding new row, updating a row, etc…) and from these events that we can subscribe, we can tell the grid to re-render but that’s kinda of a manual process, we need to subscribe and call a grid render, it won’t do it for you and like I said earlier it will only re-render what is shown in the screen (because of the Virtual Rendering), so it’s a deep binding… or at least I wouldn’t call that a deep binding and the Virtual Rendering is what makes this grid super fast and responsive.

[jeremyholt]

I don’t know if this will help you visualize better the state

If you look closely, you will see that almost every item uses the same pattern I mentioned above, current and list[].

I know the dox say you can have more than one state, but it seems to me that the whole point of aurelia-store is that everything is stored in one place only.

To give you a specific idea from the state you see in the picture …

1. A purchase of goods is made from a supplier
2. When the truck arrives at the warehouse an arrival is recorded
3. After the goods are inspected (part of the arrival) the warehouse manager moves them into stock.

Say I am on the “arrival page” and I need to display some information about the purchase, - I already have state.purchase (I know an arrival can only come from that purchase). If I need the supplier’s full name, I already have a list of suppliers (supplier.list), so I can do a quick

const name = this.state.supplier.list.find(c=>c.id==this.state.purchase.supplierId).name;

Actually the way I really “learned” about store, was because this was the first app I’ve written that is mobile only - so I have to have the list of items in one view and the details in another view. I’m constantly hopping around between the list and the details. I don’t have the physical space on the screen to show the master/details. It forced me to completely rethink what I was doing - for example sending the supplier’s name as a parameter on the route so I could display it in the arrival view.

I don’t proclaim to be an expert on aurelia-store by any means - you need @zewa666 for that

[sousekd]

@bigopon I have prepared a simple example of a form. I would like to be able to observe all “data” changes easily: https://codesandbox.io/s/complex-form-example-b63wo