Release Notes for Early Beta Releases

These release notes are for the early beta releases of Opus/UrbanSim (pre-version 4.0). These notes cover changes since July 2006.

Changes since 14 July 2006 (Opus version 1.0 beta1)

17 November 2006

We made a set of changes to dataset names:

• We now use the singular case for the dataset names and their class names.  The dataset "gridcell", for instance now is defined by class "GridcellDataset" defined in Python module "gridcell.py".  Plural is still used in the database table names, e.g. table gridcells.  Using singular everywhere except for the table names makes it easier.  (In the near future, we plan to extend the configuration so that the user specifies which module to use for each table.  This will allow the user to use the same dataset class for multiple tables, and to use whatever table names suit them, regardless of the dataset names.)
• The class names for all datasets now end with "Dataset" instead of "Set", e.g. we now use "GridcellDataset" (instead of "GridcellSet"). 
• The interaction datasets now include an "x" in their class and module names, e.g. we now define class "HouseholdXGridcellDataset" in Python module "household_x_gridcell" (instead of "HouseholdGridcellSet" in Python module "household_gridcell.py").  This makes it easier to visually see which dataset is an interaction set, and eliminates ambiguity about (a) whether a dataset such as "employment_center" is a single dataset or an interaction set and (b) where the first dataset name ends and the second name begins.

To update the specification tables in your database to use the new names, use the upgrade scripts (see opus_upgrade/changes_2006_11_16_interaction_set_changes/readme.html).

If your models use the "building_types" and "buildings" table, use the upgrade scripts located here to add these tables to your dataset: opus_upgrade/changes_2006_09_26/readme.html.

2 November 2006

The database table "opus_development_constraints" is now named "development_constraints".  Please make this name change in your database.

24 October 2006

The start_run tool now requires that stored configurations either:

1. be in files whose module name is "equivalent" its class name, where the module name is lower_case_with_underscores and its class is CamelCase, e.g. module a_stored_configuration.py would define class AStoredConfiguration.
2. or define a run_configuration object that is the configuration object.

18 October 2006

Removed update method from the Configuration class.  Instead, use the merge method.  (The problem was that the update method replaced values at the top level of the configuration, which often removed much more than desired.  The merge method only changes values at the "leaves" leaving the rest of the Configuration unchanged.  For instance, if a Configuration C contains the data:

{
 'a':{
 'b':10,
 'c':20
 }
}

then C.merge({'a':{'n':30}}) will result in:

{
 'a':{
 'b':10,
 'c':20,
 'n':30
 }
}

whereas C.update({'a':{'n':30}}) would have resulted in:

{
 'a':{
 'n':30
 }
}

16 October 2006

The create_database() method of MysqlDatabaseServer no longer has an "overwrite" argument. If you want to delete the database first, you have to do that explicitly, e.g. with a call to drop_database().

13 October 2006

Also placed 'unroll_gridcells' under 'creating_baseyear_cache_configuration'.

12 October 2006

Grouped the following configuration entries under a single 'creating_baseyear_cache_configuration' entry:

• 'baseyear_cache'
• 'cache_directory_root'
• 'cache_directory'
• 'cache_from_mysql'
• 'tables_to_cache'
• 'tables_to_cache_nchunks'
• 'tables_to_copy_to_previous_years'

11 October 2006

MysqlDatabaseServer now expects a DatabaseConfiguration or DatabaseServerConfiguration object (or similar interface).

10 October 2006

Delegated some data from DatabaseConfiguration to a DatabaseServerConfiguration.

All 'output_configuration's are now DatabaseConfiguration objects.

9 October 2006

All 'input_configuration's are now DatabaseConfiguration objects.

A safer fork_process that takes care of the pickling and cleaning up for you.

DatabaseConfiguration now requires all fields to be specified upon construction. (No more defaults.)

Opus and UrbanSim now require the Traits package (http://code.enthought.com/traits/) for some of the configurations.  This is the start of a conversion to using Trait-based configuration objects throughout Opus, which we hope to complete by the end of this month.  The Traits package is included in the Enthought Python Edition.  

Added opus_upgrade project to our installer.

6 October 2006

Changed the code so that it really does require the 'cache_mysql_data' entry in the configuration.  The Manual had already stated that this entry was mandatory, but the code was not using it (ouch!).

29 September 2006

• The NewDatabaseServer.get_database() method now returns a ScenarioDatabase object if the database requested contains a "scenario_information" table (and thus may have a non-empty chain of databases through which it should look for tables), and an OpusDatabase object otherwise.  Thus, the user no longer needs to check which type of database object to use.
• The upgrade scripts will now work with a chain of scenario databases, such as is used within CUSPA for configuration management of our "gold standard" database.

28 September 2006

Agent Relocation Model was modified in a way that if no 'probabilities' argument is passed (or its value is None), the model returns simply an index of all unplaced agents. If 'probabilities' is given, the behaviour remains unchanged, i.e. it results in an index of all unplaced agents plus an index of sampled agents.

25 September 2006

The need for having a job attribute 'home_based' was removed. The Employment Transition Model and all variables which use home_based and non_home_based distinction of jobs determine these from the job attribute 'building_type' and the column 'home_based' of the table 'job_building_types'. Thus, the column 'home_based' of the table 'jobs' becomes obsolete and can be removed.

22 September 2006

The four different employment location choice models:

• Commercial ELCM
• Industrial ELCM
• Home Based ELCM
• Scaling Jobs Model

were merged into one model group (called 'Employment Location Choice Model') with several members (submodels). This allows easy extension to other types of ELCMs.

Use the upgrade scripts located here to update your database to work with these changes: opus_upgrade/elcm_changes_2006_09_20/readme.html.

Here are more details about these changes:

1. Create a table of building types for jobs, called 'job_building_types', that determines the members of the ELCM group. It has the following structure and example values:

   id	name	home_based
   1	commercial	0
   2	governmental	0
   3	industrial	0
   4	home_based	1

   Such table can be created using the script do_create_job_building_types_table_for_elcm.py.
2. If the submodels are using specification and coefficient tables, these must be called type_employment_location_choice_model_specification/coefficients where type stands for the corresponding name from the name column in the above table. Since until now we used tables called employment_type_location_choice_model_specification/coefficients, they must be renamed. This can be done for types commercial, industrial and home_based using the script do_rename_by_swapping_employment_and_commercial_or_industrial_or_home_based_for_elcm.py.
3. If you have separate tables of jobs for estimation for the different submodels, such as jobs_for_estimation_commercial, jobs_for_estimation_industrial, ..., put them into one table, called e.g. 'jobs_for_estimation'. You can use the script do_combine_job_for_estimation_tables_for_elcm.py.

The configuration for running UrbanSim is by default set up to run all submodels found in the table above. I.e. entry 'models' in general_configuration.py contains:

{"employment_location_choice_model": {"group_members": "_all_"}}

This can be changed to specific submodels, e.g. by:

{"employment_location_choice_model": {"group_members": ["commercial", "industrial"]}}

Each submodel selects the appropriate jobs according to values in the attribute 'building_type' of the jobs table. Therefore, we are not running the Divide Jobs Model anymore.

All submodels with one exception are configured by default as location choice models. The configuration can be found in the controller entry for "employment_location_choice_model". One can overwrite this settings for a specific submodel with an entry "type_employment_location_choice_model" which has a priority over the default controller. This is done for the governmental submodel which is running the Scaling Jobs Model.

We added a new model, called Distribute Unplaced Jobs Model. This model determines all unplaced jobs in the job set and places them using the Scaling Jobs Model. This assures that all jobs will be placed, even if they are not explicitly included in the specification of one of the ELCM models.

20 September 2006

Changed all configurations so that 'db_input_database', 'db_host_name', 'db_password', and 'db_output_database' are specified under 'input_configuration' or 'output_configuration', as appropriate.

19 September 2006

Removed the opus_core.tools.get_status command, since it never did much of use and we have a better design for how to do this but don't plan to implement it soon.  Also removed the rest of the associated heuristics and heartbeat code.

14 September 2006

Change semantics of Configuration's merge() method.  Before, for the command to_config.merge(from_config) if from_config had an entry that was a dictionary, then that dictionary would *replace* the same-named entry in to_config.  Now, the individual values from the dictionary in from_config will be added to the same-named entry in to_config.  This allows from_config to contain a set of "spot changes" of values to insert into the specified locations within to_config.

Renamed dataset.set to dataset.attribute_boxes.

Moved AttributeBox into opus_core.attribute_box.

Removed heuristics, RunMonitor and fork_new_monitored_process from the system.

Added is_invalid method to check for NaNs in Coefficients. estimator_new will no longer try to save results that cause NaNs.

13 September 2006

Renamed and moved opus_core.services.run_server.misc to opus_core.fork_process after moving create_datasets_from_flt into opus_core.dataset_factory.

7 September 2006

The sampling rates database constants were not being used in the code. They have been removed.

6 September 2006

Replaced the use of the translate* methods of Resources & GeneralResources with the equivalent standard dict syntax, since it uses the Python standard and is more concise.

4 September 2006

Moved opus_core.services.run_server.create_flt_cache to opus_core.store.create_flt_cache.

Moved opus_core.services.run_server.create_baseyear_cache to opus_core.tools.create_baseyear_cache.

Moved opus_core.services.run_server.create_services_database to opus_core.tools.create_services_database.

Moved opus_core.services.run_server.delete_run to opus_core.tools.delete_run.

Moved opus_core.services.run_server.get_status to opus_core.tools.get_status.

Moved opus_core.services.run_server.restart_run to opus_core.tools.restart_run.

Moved opus_core.services.run_server.scenario_database to opus_core.store.scenario_database.

1 September 2006

Finished implementing automatic unit test detection and running for all_tests.

Added a nicer looking testRunner for all_tests: opus_core.tests.utils.opus_test_runner. When used, OpusTestRunner will display the name and module of each test as it is being run.

31 August 2006

Lag variables: Fixed a bug with lag variables.  For lag variables for computed attributes (those defined by an Opus variable) the code had been always returning the values from the current year.  Now it returns the correct values from the correct prior year.  

Unrolling gridcells: Fixed a bug in the processing of the development_event_history table.  The bug was that when converting the year N data to year N-1, the code had been using the development events for year N-1, but should have been using the events for year N.

Sometimes a lag variable will need to compute data for before the base year, and that computation will require some of the "invariant" data that was copied from the baseyear database into the baseyear cache.  If this is the case, add those database tables to the list of tables in the configuration's 'tables_to_copy_to_previous_years' entry.

Variables computing "recent" changes: Changed the semantics of whether to look in the current year when computing the number or amount of "recent" changes, e.g. for urbansim.gridcell.n_recent_development_projects.  The code had been including the items from the current year as well as the prior N years.  It now only looks at items from the prior N years.  This provides a more consistent view of "recent" changes, since even if additional changes occur during the current year, models before and after those changes will still see the same values for the variables on "recent" changes.  We changed the following two variable definitions, though other variables that depend upon them also now produce different results: 

• urbansim.gridcell.n_recent_development_projects
• urbansim.gridcell.n_recent_transitions_to_developed

30 August 2006

Types of attributes: Changed 'secondary' attributes (formerly known as 'derived' attributes) to be 'computed' attributes.

25 August 2006

Finished replacing "entity" with "dataset", since they were synonyms.

Added 'cache_mysql_data' entry (the full Opus path to a cache_mysql_data model) to the urbansim run_configuration and changed the opus_core create_baseyear_cache tool to expect such an entry and act appropriately.

24 August 2006

Removed the datasets and session_configuration code from run_server.run_manager. Removed the opus_core model_system. Added a 'model_system' argument to the urbansim base_config run_configuration to specify the location of the model system to use (e.g. 'urbansim.model_coordinators.model_system'). Removed the --datasets_module option from run_server restart_run.py, start_run_set.py, start_run.py.

23 August 2006

Converted many references of "entity" to "dataset." At this point, most occurrences of "entity_name" have been changed, but no others.

Moved:

• biocomplexity.utils.misc to biocomplexity.misc and removed the utils folder.
• opus_core.utils.file_utilitites to opus_core.file_utilities.
• opus_core.utils.latex to opus_core.latex and removed the utils folder.
• randstad.utils.gridcell_classifier to randstad.gridcell_classifier.
• urbansim.utils.flatten_scenario_database_chain to urbansim.flatten_scenario_database_chain.
• urbansim.utils.latex_table_creator to urbansim.latex_table_creator.
• urbansim.utils.rollback_gridcells to urbansim.rollback_gridcells.
• urbansim.utils.synthesize_jobs to urbansim.synthesize_jobs and removed the utils directory.

21 August 2006

We completed the move of the 'services' Opus package into opus_core. Because the urbansim dependencies were removed, opus_core.services.run_server.run_manager run_run() and restart_run() now require a 'datasets' argument (such as 'urbansim') specifying in which module the datasets are kept. Added a --datasets_module option to run_server restart_run.py, start_run_set.py, start_run.py.

14 - 18 August 2006

We renamed the 'emme2' Opus package to be "opus_emme2."

We renamed the 'manual' Opus package to be "opus_manual."

We changed 'non-derived' attributes to be 'primary' attributes and 'derived' attributes to be 'secondary' attributes. (It looks like this might change again...)

More indexing of the manual was done.

11 August 2006

We just finished two major changes to the Opus code and data that will require you to completely update your Opus installation and change some of your baseyear data :

• We renamed the "core" Opus package to be "opus_core".
• We removed the "opus" directory and all "opus." references from our Opus system.
  • All Opus packages are now top-level Python packages.
  • The "opus" directory is gone.
  • We no longer use the "opus." in any of our imports or our fully-qualified Opus paths.
  • Variable specifications no longer include "opus." at the beginning; instead of typing:

  opus.core.func.ln(opus.urbansim.gridcell.residential_units)

  you should now type:

  ln(urbansim.gridcell.residential_units)

To update to this new system:

1. Uninstall your current Opus installation, if it is from before 11 August 2006.
2. Install an Opus version from 11 August 2006 or after.
3. Remove any instance of the sub-string "opus." from your Python code.
4. Use the database conversion tool opus_upgrade/changes_2006_08_11/using_convert_databases.html to change your variable_name entries in your database specification tables (e.g. in land_price_model_specification).  This tool will:
   • Remove any instance of the sub-string "opus." 
   • Change any instance of "core" to "opus_core"

Minor changes

• Created a place for user contributed items.
• Documented steps necessary to convert an UrbanSim 3 baseyear database into the form needed for UrbanSim 4.
• Moved much of services into opus_core (and are in process of moving the rest of it).
• The manual Opus package was renamed to opus_manual.
• The emme2 Opus package was renamed to opus_emme2.