Release Notes for Version 4.0 and earlier beta releases

These release notes are for the 4.0 versions of Opus/UrbanSim,. These notes cover changes since 21 November 2006.

Changes since 21 November 2006 (Opus version 4.0)

16 April 2007

The Opus classes Coefficients and EquationSpecification were modified in a way that they handle their class attributes as arrays (and not as tuples anymore). These changes are mainly internal; the constructors still work with arguments passed as tuples. Nevertheless, some methods now return arrays instead of tuples.

10 April 2007 - Converting from numarray to numpy

Opus and UrbanSim used to use numarray.  They now use numpy.  This change was done for several reasons:

  • numarray is being deprecated
  • numpy is better designed than numarray
  • numpy runs faster than numarray (for instance, on an Intel Mac our simulations now run 1/3 faster with numpy)

This change is extensive, involving many thousands of changes to the code.  Some of the important externally visible changes are:

  • You need to install (or upgrade) numpy and install scipy.  See the installation instructions for your system.
    • If you already have numpy, make sure it is version 1.0.1 or greater.  If not, please upgrade it.
    • If you already have scipy, make sure it is version 0.5.2 or greater.  If not, please upgrade it.
  • Opus flt_storage files (and thus attribute_cache files) are stored differently with different file names:
    • The binary data is stored in a different format (numpy instead of numarray).    
    • The flt_storage files (and thus attribute_cache files) have different file extensions. For instance, a file that used to have file extension 'Float32' now has file extension 'lf4' or 'bf4', depending upon whether your computer's processor uses little-endian ('l') or big-endian ('b'). These file extensions correspond to the numpy dtype values, except that the endian symbols have been converted from '<', '>', or '|' to 'l', 'b', or 'i' (for little-endian, big-endian, or irrelevant-endian).
    • Text attributes are stored in a fixed-width format. The file extension includes the number of characters used for each value.  For instance, an extension of '.iS12' uses 12 characters for each value.
  • Attribute caches produced by the numarray version of Opus will not work with the numpy-based Opus.  However, we do provide a cache conversion tool that will create a new cache containing all of the data as an old cache but stored with the new format and file names (see opus_upgrade/changes_2007_04_11/readme.html).

Internally, the code has been changed in many, many ways (see Section 2.7 of Travis Oliphant's Guide to Numpy, and Christopher Hanley's notes).  Some significant changes not in these notes are:

  • In numpy the functions "ones" and "zeros" by default create arrays with data of type float32.  In numarray the default data type is int.  To get around this, in numpy you can use "ones_like" and "zeros_like" on an array of integers, or you can explicitly specify the dtype='i' argument in the ones or zeros, e.g. ones(3, dtype='i') produces an array of three integers.
  • In numpy, assigning an array of a single element into a single element of an array of floats will produce the error "TypeError: array cannot be safely cast to required type".  In numarray, the single-valued-array is automatically be cast into a single value and put into the destination array. 
  • In numpy, invoking concatenate an array of strings and an array of integers, for instance, returns an array of strings. In numarray concatenate fails on such mis-matched types.
  • The functions numarray.stddev and numarray.nd_image.standard_deviation produce different results from the numpy.std function. Similarly, the numarray.nd_image.variance function produces different results from  numpy.var function. Numpy uses biased computations: 
    1/N(sum((x-mu)^2))

    whereas numarray uses unbiased computations: 

1/(N-1)(sum((x-mu)^2))

This does not make much difference for larger arrays, but can be quite different for very small arrays.  The numpy.oldnumeric.mlab.std function also uses biased computations. If you want unbiased computation, use scipy.ndimage.variance and scipy.ndimage.standard_deviation. 

  • The axis parameter of take is more restrictive in numpy.
  • The numpy.argmax function requires an axis argument for a 2D array.  The numarray.argmax function does not.
  • Function put behaves differently. Instead of
put(myarray, (index1, index2), values, axis=(0,1))
use 
myarray[index1, index2] = values
  • Functions of numpy.random for drawing random numbers do not take arrays as arguments for distribution parameters.
  • In numpy, arrays of type float96 and float128 print and display incorrectly, even though the data is correct.  This is a problem in numpy version 1.0.2, and will be fixed in the next numpy release.
  • numpy.random.seed takes at most one parameter, as opposed to numarray.random_array.seed, which takes up to two inputs.

22 March 2007

The way expressions for Opus variables are defined has been greatly enhanced. See Section 6.3 (Opus Variables) in Chapter 6, and Section 7.4 (Opus Variables) in Chapter 7 of the user manual for information. Existing references to variable names should continue to work without change, but any expressions or aliases will need to be updated. Here is a list of things that may need to be changed in user variable definitions:

  • Alias declarations.  Example: 
        urbansim.gridcell.population as pop      
    should be replaced by 
        pop = urbansim.gridcell.population

  • Function applications.  Example: 
        opus_core.func.ln(urbansim.gridcell.population)
    should be replaced by 
        ln(urbansim.gridcell.population)

  • Aggregation and disaggregation operations.  Examples:

        district:opus_core.func.aggregate(gridcell.population, [zone, faz])
    should be replaced by
        district.aggregate(gridcell.population, intermediates=[zone, faz])

        opus_core.func.aggregate(gridcell.population, [], mean)
    should be replaced by
        zone.aggregate(gridcell.population, function=mean)
    assuming this is being aggregated to the zone level

Changing the function names (e.g. opus_core.func.ln to ln) can probably be done with a search-and-replace in all the relevant files, since the function names themselves haven't changed -- just remove the opus_core.func prefix. Aliases will probably require hand editing but are routine. (The only exception would be if you have declared an alias for an expression embedded inside another expression. In that case you'll need to break it into two expressions, since aliases are now declared using an assignment statement, which must be at the top level. You probably don't have any embedded aliases though ....)

Updating the aggregation and disaggregation will require more hand editing, since these are now syntactically methods -- you'll need to put the receiver dataset in front of the aggregate or disaggregate method call. Also, the optional parameters should now use keywords ("intermediates" and "function").

These changes to variable names will probably require that models be re-estimated if the new version of Opus is used.

25 February 2007

There have been significant changes to the indicator framework, which is now part of the opus_core. The major change has been a change from using dictionaries to specify indicators to using objects to specify indicators. The new indicator API is described in the "Generating and Visualizing Indicators" chapter of the opus manual.Old batch indicator scripts need to be updated to support the new indicator framework. Please contact us if there are problems updating.

13 February 2007

Replaced tab_storage and csv_storage with new modules that support the built-in Python csv module, rather than use custom code.

31 January 2007

Removed the DbConnection class, all instances are now ScenarioDatabase or OpusDatabase, depending on whether linking between databases is required.

3 January 2007

Eliminated ram_storage.

Removed the data parameter from Dataset. Now requires an in_storage object.

22 December 2006

mysql_storage no longer maintains a database connection, but rather constructs one only for the short period of time it is necessary, closing it immediately after. Parameters have been added to the constructor such that one may either pass in a database connection like before or specify the database name, hostname, username, and password instead. Once all mysql_storage objects are constructed without a database connection object, that parameter is to be removed.

Split urbansim.indiactors.indicator_maker into two files. The class remains in the same file. The script used for running was moved to urbansim.tools.make_indicators_tool.

Moved opus_core.storage to opus_core.store.storage.

21 December 2006

The SessionConfiguration object now uses a DatasetPool object to hold the set of active datasets.  If the dataset pool does not have the requested dataset, it will use the information in the new "dataset_pool_configuration" to find the module containing the requested dataset class.  If the "package_order_exceptions" has an entry for the requested dataset name, it will use the module specified.  Otherwise, it will look for a module for the requested dataset in the "datasets" directories of the Opus packages listed in the "package_order" entry of the "dataset_pool_configuration".  

Configurations for simulation now require an entry for "dataset_pool_configuration".  Please add one if it does not have one.

Replaced the use of the Constants class with UrbansimConstantDataset - a dataset-like class that contains all of the values defined in the "urbansim_constants" table as well as some values derived from those.  (Actually, at the moment UrbansimConstantDataset is a facade to Constants.)  This dataset now acts like a read-only dataset, so it can be constructed via DatasetPool (via DatasetFactory) just like any other dataset.  

Changed all configurations that used parameter "constants" to use parameter "urbansim_constant".

The "urbansim_constants" must be copied to the earliest year unrolled.  Please add an entry such as

    'urbansim_constants': 1975,

to your configuration's "tables_to_copy_to_previous_years" entry, where the year listed is the earliest year unrolled.

Created an OpusTestCase class that automatically destroys all singletons before and after each unit test, in order to ensure that each unit test has a clean state.

Created test and test_x_test datasets for unit testing in opus_core.

The tab_storage now requires its tab files to be in files with an extension of ".tab".  The "in_table_name" argument to tab_storage should not have any file extension, e.g. it should be like "urbansim_constant"  (not "urbansim_constant.tab"), so that the same in_table_name will work with other storage objects.

Cleaned up the washtenaw Opus package by moving some files that were used for the UrbanSim course, but are no longer how we do things, into the washtenaw directory of the sandbox Opus package.

20 December 2006

Renamed directory_to_cache to existing_cache_to_copy, since it was difficult to intuit the difference between cache_directory and directory_to_cache.

Removed the is_low_memory_run parameter from AttributeCache. Use the one in SimulationState instead.

19 December 2006

Moved opus_core.attribute_cache to opus_core.store.attribute_cache

12 December 2006

All storage objects are now constructed actively before being passed to the system rather than passively through specifying in/out_storage_type and in/out_storage_location. (There is no more guessing the parameters for constructing a storage object deep down in the code.) Thus, in/out_storage_type and location have been removed throughout the system, as they are no longer needed. As a consequence, when constructing Datasets, specifying configurations, and similar, in/out_storage will need to be specified directly, rather than specifying the storage type and location, as those parameters no longer exist.

11 December 2006

Moved opus_core.dataset to opus_core.datasets.dataset.

7 December 2006

Changed SessionConfiguration to require an in_storage parameter when creating a new instance.

Removed references to no-longer-used "change_type" fields in development_events_history and development_events_exogenous.

6 December 2006

Removed the resources argument from the opus_core.store storage classes' __init__ methods and StorageFactory.

5 December 2006

StorageCreator was almost exactly a duplicate of StorageFactory, so it has been removed. StorageFactory is used instead. For the time being, storage types such as 'ram' and 'mysql' have been replaced with 'ram_storage' and the like in order to get rid of StorageCreator. StorageCreator was not the proper place for such a conversion (of 'ram' to 'ram_storage', for example), and this functionality needs to be written elsewhere, later, if at all.

4 December 2006

The estimation process no longer uses a creating_baseyear_cache_configuration.

Moved cache_directory out of creating_baseyear_cache_configuration.

30 November 2006

Changed names of the dataset Python modules to to use singular with a "_dataset" suffix, e.g. we now use "gridcell_dataset.py" (instead of "gridcells.py").

29 November 2006

Replaced the contents of urbansim.configs.general_configuration with the contents from urbansim.configs.general_configuration_with_changed_elcm (and then deleted general_configuration_with_changed_elcm).

28 November 2006

Changed all 'creating_baseyear_cache_configuration' entries to use opus_core- or urbansim.configurations.creating_baseyear_cache_configuration. Flattened each resulting entry to account for inability to merge non-dictionary objects in the code.

Added urbansim.configurations.creating_baseyear_cache_configuration. Adds an unroll_gridcells parameter and changes the default of cache_mysql_data to 'urbansim.model_coordinators.cache_mysql_data'.

Changed opus_core.configurations.creating_baseyear_cache_configuration to match the current state of our configurations.

27 November 2006

Changed all 'baseyear_cache' configuration entries to use opus_core.configurations.baseyear_cache_configuration.

Changed case of class 'DataSet' to be 'Dataset'.