Persistence¶

On the Principles of DDD¶

From Evans, the rules we need to enforce include:

• The root Entity has global identity and is ultimately responsible for checking invariants.
• Root Entities have global identity. Entities inside the boundary have local identity, unique only within the Aggregate.
• Nothing outside the Aggregate boundary can hold a reference to anything inside, except to the root Entity. The root Entity can hand references to the internal Entities to other objects, but they can only use them transiently (within a single method or block).
• Only Aggregate Roots can be obtained directly with database queries. Everything else must be done through traversal.
• Objects within the Aggregate can hold references to other Aggregate roots.
• A delete operation must remove everything within the Aggregate boundary all at once.
• When a change to any object within the Aggregate boundary is committed, all invariants of the whole Aggregate must be satisfied.

On the relationship between adding and retrieving¶

When you add() something to a repository and do a findAll() immediately afterwards, you might be surprised: the freshly added object will not be found. This is not a bug, but a decision we took on purpose. Here is why.

When you add an object to a repository, it is added to the internal identity map and will be persisted later (when persistAll() is called). It is therefore still in a transient state - but all query operations go directly to the underlying data storage, because we need to check that anyway. So instead of trying to query the in-memory objects we decided to ignore transient objects for queries [1].

If you need to query for objects you just created, feel free to have the PersistenceManager injected and use persistAll() in your code.

How changes are persisted¶

When you add or remove an object to or from a repository, the object will be added to or removed from the underlying persistence as expected upon persistAll. But what about changes to already persisted objects? As we have seen, those changes are only persisted, if the changed object is given to update on the corresponding repository.

Now, for objects that have no corresponding repository, how are changes persisted? In the same way you fetch those objets from their parent - by traversal. FLOW3 follows references from objects managed in a repository (aggregate roots) for all persistence operations, unless the referenced object itself is an aggregate root.

When using the Doctrine 2 persistence, this is done by virtually creating cascade attributes on the mapped associations.

Requirements and restrictions¶

There are some rules imposed by Doctrine (and/or FLOW3) you need to follow for your entities (and value objects). Most of them are good practice anyway, and thus are not really restrictions.

• Entity classes must not be final or contain final methods.
• Persistent properties of any entity class should always be protected, not public, otherwise lazy-loading might not work as expected.
• Implementing __clone() or __wakeup() is not a problem with FLOW3, as the instances always have an identity. If using your own identity properties, you must wrap any code you intend to run in those methods in an identity check.
• Entity classes in a class hierarchy that inherit directly or indirectly from one another must not have a mapped property with the same name.
• Entities cannot use func_get_args() to implement variable parameters. The proxies generated by Doctrine do not support this for performance reasons and your code might actually fail to work when violating this restriction.

Persisted instance variables must be accessed only from within the entity instance itself, not by clients of the entity. The state of the entity should be available to clients only through the entity’s methods, i.e. getter/setter methods or other business methods.

Collection-valued persistent fields and properties must be defined in terms of the Doctrine\Common\Collections\Collection interface. The collection implementation type may be used by the application to initialize fields or properties before the entity is made persistent. Once the entity becomes managed (or detached), subsequent access must happen through the interface type.

Metadata mapping¶

The Doctrine 2 ORM needs to know a lot about your code to be able to persist it. Natively Doctrine 2 supports the use of annotations, XML, YAML and PHP to supply that information. In FLOW3, only annotations are supported, as this aligns with the philosophy behind the framework.

Differences between FLOW3 and plain Doctrine¶

The custom annotation driver used by FLOW3 to collect mapping information from the code makes a number of things easier, compared to plain Doctrine 2.

• Entity

  • repositoryClass can be left out, if you follow the naming rules for your repository classes explained above.

• Table

  • name does not default to the unqualified entity classname, but a name is generated from classname, package key and more elements to make it unique.

• Id

  • Can be left out, as it is automatically generated, this means you also do not need @GeneratedValue. Every entity will get a property injected that is filled with an UUID upon instantiation and used as technical identifier.
  • If an @Id annotation is found, it is of course used as is and no magic will happen.

• Column

  • Can usually be left out altogether, as the vital type information can be read from the @var annotation on a class member.

    Important

    Since PHP does not differentiate between short and long strings, but databases do, you must use @Column(type="text") if you intend to store more than 255 characters in a string property.

• OneToOne

• OneToMany

• ManyToOne

• ManyToMany

  • targetEntity can be omitted, it is read from the @var annotation on the property

• JoinTable

• JoinColumn

  • Can usually be left out completely, the needed information is gathered automatically

  • But when using a self-referencing association, you will need to help FLOW3 a little, so it doesn’t generate a join table with only one column.

    Example: JoinTable annotation for a self-referencing annotation:

            /**
             * @var \Doctrine\Common\Collections\ArrayCollection<\TYPO3\Blog\Domain\Model\Post>
             * @ORM\ManyToMany
             * @ORM\JoinTable(inverseJoinColumns={@ORM\JoinColumn(name="related_id")})
             */
             protected $relatedPosts;
    
    Without this, the created table would not  contain two columns but only one, named
    after the identifiers of the associated entities - which is the same in this case.
    

• DiscriminatorColumn

• DiscriminatorMap

  • Can be left out, as they are automatically generated.

The generation of this metadata is slightly more expensive compared to the plain Doctrine AnnotationDriver, but since this information can be cached after being generated once, we feel the gain when developing outweighs this easily.

Here is an example to illustrate the things you can omit, due to the automatisms in the FLOW3 annotation driver.

Example: Annotation equivalents in FLOW3 and plain Doctrine 2

An entity with only the annotations needed in FLOW3:

/**
 * @FLOW3\Entity
 */
class Post {

  /**
   * @var \TYPO3\Blog\Domain\Model\Blog
   * @ORM\ManyToOne(inversedBy="posts")
   */
  protected $blog;

  /**
   * @var string
   * @ORM\Column(length=100)
   */
  protected $title;

  /**
   * @var \DateTime
   */
  protected $date;

  /**
   * @var string
   * @ORM\Column(type="text")
   */
  protected $content;

  /**
   * @var \Doctrine\Common\Collections\ArrayCollection<\TYPO3\Blog\Domain\Model\Comment>
   * @ORM\OneToMany(mappedBy="post")
   * @ORM\OrderBy({"date" = "DESC"})
   */
  protected $comments;

The same code with all annotations needed in plain Doctrine 2 to result in the same metadata:

/**
 * @ORM\Entity(repositoryClass="TYPO3\Blog\Domain\Model\Repository\PostRepository")
 * @ORM\Table(name="blog_post")
 */
class Post {

  /**
   * @var string
   * @ORM\Id
   * @ORM\Column(name="flow3_persistence_identifier", type="string", length=40)
   */
  protected $FLOW3_Persistence_Identifier;

  /**
   * @var \TYPO3\Blog\Domain\Model\Blog
   * @ORM\ManyToOne(targetEntity="TYPO3\Blog\Domain\Model\Blog", inversedBy="posts")
   * @ORM\JoinColumn(name="blog_blog", referencedColumnName="flow3_persistence_identifier")
   */
  protected $blog;

  /**
   * @var string
   * @ORM\Column(type="string", length=100)
   */
  protected $title;

  /**
   * @var \DateTime
   * @ORM\Column(type="datetime")
   */
  protected $date;

  /**
   * @var string
   * @ORM\Column(type="text")
   */
  protected $content;

  /**
   * @var \Doctrine\Common\Collections\ArrayCollection<\TYPO3\Blog\Domain\Model\Comment>
   * @ORM\OneToMany(targetEntity="TYPO3\Blog\Domain\Model\Comment", mappedBy="post",
    cascade={"all"}, orphanRemoval="true")
   * @ORM\OrderBy({"date" = "DESC"})
   */
  protected $comments;

Query the schema status¶

To learn about the current schema and migration status, run the following command:

$ ./flow3 flow3:doctrine:migrationstatus

This will produce output similar to the following, obviously varying depending on the actual state of schema and active packages:

Example: Migration status report

== Configuration
   >> Name:                                               Doctrine Database Migrations
   >> Database Driver:                                    pdo_mysql
   >> Database Name:                                      flow3
   >> Configuration Source:                               manually configured
   >> Version Table Name:                                 flow3_doctrine_migrationstatus
   >> Migrations Namespace:                               TYPO3\FLOW3\Persistence\Doctrine\Migrations
   >> Migrations Target Directory:                        /path/to/Data/DoctrineMigrations
   >> Current Version:                                    0
   >> Latest Version:                                     2011-06-13 22:38:37 (20110613223837)
   >> Executed Migrations:                                0
   >> Available Migrations:                               1
   >> New Migrations:                                     1

== Migration Versions
   >> 2011-06-13 22:38:37 (20110613223837)                not migrated

Whenever a version number needs to be given to a command, use the short form as shown in parentheses in the output above. The migrations directory in the output is only used when creating migrations, see below for details on that.

Deploying migrations¶

On a pristine database it is very easy to create the tables needed with the following command:

$ ./flow3 flow3:doctrine:migrate

This will result in output that looks similar to the following:

Migrating up to 20110613223837 from 0

  ++ migrating 20110613223837

     -> CREATE TABLE flow3_resource_resourcepointer (hash VARCHAR(255) NOT NULL, ⏎
     PRIMARY KEY(hash)) ENGINE = InnoDB
     -> ALTER TABLE flow3_resource_resource ADD FOREIGN KEY ⏎
     (flow3_resource_resourcepointer) REFERENCES flow3_resource_resourcepointer(hash)

  ++ migrated (1.31s)

  ------------------------

  ++ finished in 1.31
  ++ 1 migrations executed
  ++ 6 sql queries

This will deploy all migrations delivered with the currently active packages to the configured database. During that process it will display all the SQL statements executed and a summary of the deployed migrations at the and. You can do a dry run using:

$ ./flow3 flow3:doctrine:migrate --dry-run

This will result in output that looks similar to the following:

Executing dry run of migration up to 20110613223837 from 0

  ++ migrating 20110613223837

     -> CREATE TABLE flow3_resource_resourcepointer (hash VARCHAR(255) NOT NULL, ⏎
     PRIMARY KEY(hash)) ENGINE = InnoDB
     -> ALTER TABLE flow3_resource_resource ADD FOREIGN KEY ⏎
     (flow3_resource_resourcepointer) REFERENCES flow3_resource_resourcepointer(hash)

  ++ migrated (0.09s)

  ------------------------

  ++ finished in 0.09
  ++ 1 migrations executed
  ++ 6 sql queries

to see the same output but without any changes actually being done to the database. If you want to inspect and possibly adjust the statements that would be run and deploy manually, you can write to a file:

$ ./flow3 flow3:doctrine:migrate --path <write/here/the.sql>

This will result in output that looks similar to the following:

Writing migration file to "<write/here/the.sql>"

Reverting migrations¶

The migrate command takes an optional --version option. If given, migrations will be executed up or down to reach that version. This can be used to revert changes, even completely:

$ ./flow3 flow3:doctrine:migrate --version <version> --dry-run

This will result in output that looks similar to the following:

Executing dry run of migration down to 0 from 20110613223837

  -- reverting 20110613223837

     -> ALTER TABLE flow3_resource_resource DROP FOREIGN KEY
     -> DROP TABLE flow3_resource_resourcepointer
     -> DROP TABLE flow3_resource_resource
     -> DROP TABLE flow3_security_account
     -> DROP TABLE flow3_resource_securitypublishingconfiguration
     -> DROP TABLE flow3_policy_role

  -- reverted (0.05s)

  ------------------------

  ++ finished in 0.05
  ++ 1 migrations executed
  ++ 6 sql queries

Executing or reverting a specific migration¶

Sometimes you need to deploy or revert a specific migration, this is possible as well.

$ ./flow3 flow3:doctrine:migrationexecute --version <20110613223837> --direction <direction> --dry-run

This will result in output that looks similar to the following:

-- reverting 20110613223837

   -> ALTER TABLE flow3_resource_resource DROP FOREIGN KEY
   -> DROP TABLE flow3_resource_resourcepointer
   -> DROP TABLE flow3_resource_resource
   -> DROP TABLE flow3_security_account
   -> DROP TABLE flow3_resource_securitypublishingconfiguration
   -> DROP TABLE flow3_policy_role

-- reverted (0.41s)

As you can see you need to specify the migration --version you want to execute. If you want to revert a migration, you need to give the --direction as shown above, the default is to migrate “up”. The --dry-run and and --output options work as with flow3:doctrine:migrate.

Creating migrations¶

Migrations make the schema match when a model changes, but how are migrations created? The basics are simple, but rest assured that database details and certain other things make sure you’ll need to practice... The command to scaffold a migration is the following:

$ ./flow3 flow3:doctrine:migrationgenerate

This will result in output that looks similar to the following:

Generated new migration class to "/path/to/Data/DoctrineMigrationsVersion20110624143847.php".

Looking into that file reveals a basic migration class already filled with the differences detected between the current schema and the current models in the system:

Example: Migration generated based on schema/model differences

namespace TYPO3\FLOW3\Persistence\Doctrine\Migrations;

use Doctrine\DBAL\Migrations\AbstractMigration,
  Doctrine\DBAL\Schema\Schema;

/**
 * Auto-generated Migration: Please modify to your need!
 */
class Version20110624143847 extends AbstractMigration {

  /**
   * @param Schema $schema
   * @return void
   */
  public function up(Schema $schema) {
      // this up() migration is autogenerated, please modify it to your needs
    $this->abortIf($this->connection->getDatabasePlatform()->getName() != "mysql");

    $this->addSql("CREATE TABLE party_abstractparty (…) ENGINE = InnoDB");
  }

  /**
   * @param Schema $schema
   * @return void
   */
  public function down(Schema $schema) {
      // this down() migration is autogenerated, please modify it to your needs
    $this->abortIf($this->connection->getDatabasePlatform()->getName() != "mysql");

    $this->addSql("DROP TABLE party_abstractparty");
  }
}

To create an empty migration skeleton, pass --diff-against-current 0 to the command.

After you generated a migration, you will probably need to clean up a little, as there might be differences being picked up that are not useful or can be optimized. An example is when you rename a model: The migration will drop the old table and create the new one, but what you want instead is to rename the table. Also you must to make sure each finished migration file only deals with one package and then move it to the Migrations directory in that package. This way different packages can be mixed and still a reasonable migration history can be built up.

Schema updates without migrations¶

Migrations are the recommended and preferred way to bring your schema up to date. But there might be situations where their use is not possible (e.g. no migrations are available yet for the RDBMS you are using) or not wanted (because of, um… something). The there are two simple commands you can use to create and update your schema.

To create the needed tables you can call ./flow3 flow3:doctrine:create and it will create all needed tables. If any target table already exists, an error will be the result.

To update an existing schema to match with the current mapping metadata (i.e. the current model structure), use ./flow3 flow3:doctrine:update to have missing items (fields, indexes, ...) added. There is a flag to disable the safe mode used by default. In safe mode, Doctrine tries to keep existing data as far as possible, avoiding lossy actions.

Switching to Generic Persistence¶

To switch back to Generic persistence on SQLite using PDO you need to configure FLOW3 like this.

Objects.yaml:

TYPO3\FLOW3\Persistence\PersistenceManagerInterface:
  className: 'TYPO3\FLOW3\Persistence\Generic\PersistenceManager'

TYPO3\FLOW3\Persistence\QueryResultInterface:
  scope: prototype
  className: 'TYPO3\FLOW3\Persistence\Generic\QueryResult'

Settings.yaml:

FLOW3:
  persistence:
    doctrine:
      enable: FALSE
    backendOptions:
      dataSourceName: 'sqlite:%FLOW3_PATH_DATA%Persistent/Objects.db'
      username: ''
      password: ''
      # set the following to null to have them ignored
      driver: ''
      path: ''
      dbname: ''

Using different database systems is possible, as long as there is a PDO driver available in PHP. The syntax to use for dataSourceName depends on the PDO driver used, consult the PHP documentation for that.

When installing other backend packages, like CouchDB, the needed object configuration should be contained in them, for the connection settings, consult the package’s documentation.

Metadata mapping¶

The persistence layer needs to know a lot about your code to be able to persist it. In FLOW3, the needed data is given in the source code through annotations, as this aligns with the philosophy behind the framework.

Enabling Lazy Loading¶

If a class should be able to be lazy loaded by the PDO backend, you need to annotate it with @lazy in the class level docblock. This is done to avoid creating proxy classes for objects that should never be lazy loaded anyway. As soon as that annotation is found, AOP is used to weave lazy loading support into your code that intercepts all method calls and initializes the object before calling the expected method. Such a proxy class is a subclass of your class, as such it work fine with type hinting and checks and can be used the same way as the original class.

To actually mark a property for lazy loading, you need to add the @lazy annotation to the property docblock in your code. Then the persistence layer will skip loading the data for that object and the object properties will be thawed when the object is actually used.

How @lazy annotations interact

Schema management¶

For the PDO backend that comes with FLOW3, the needed tables are set up automatically. When models are changed, no adjustments to the schema are needed. Effectively the schema is maintenance-free. If you ever need to create or fix the schema manually, have a look at Resources/Private/Persistence/SQL/DDL.sql in the FLOW3 package.

Whether other backends implement automatic schema management is up to the developers, consult the documentation of the relevant backend for details.

Inside the Generic Persistence¶

To the domain code the persistence handling transparent, aside from the need to add a few annotations. The custom repositories are a little closer to the inner workings of the framework, but still the inner workings are very invisible. This is how it is supposed to be, but a little understanding of how persistence works internally can help understand problems and develop more efficient client code.

Persisting a Domain Object¶

After an object has been added to a repository it will be seen when FLOW3 calls persistAll() at the end of a script run. Internally all instances implementing the \TYPO3\FLOW3\Persistence\RepositoryInterface will be fetched and asked for the objects they hold. Those will then be handed to the persistence backend in use and processed by it.

FLOW3 defines interfaces for persistence backends and queries, the details of how objects are persisted and queried are up to the persistence backend implementation. Have a look at the documentation of the respective package for more information. The following diagram shows (most of) the way an object takes from creation until it is persisted when using the FLOW3 default backend:

Object persistence process

Keep in mind that the diagram omits some details like dirty checking on objects and how exactly objects and their properties are stored.

Querying the Storage Backend¶

As we saw in the introductory example there is a query mechanism available that provides easy fetching of objects through the persistence framework. You ask for instances of a specific class that match certain filters and get back an array of those reconstituted objects. Here is a diagram of the internal process when using the FLOW3 default backend:

Object querying and reconstitution process

For the developer the complexity is hidden between the query’s execute() method and the array of objects being returned.

---

[1]	An alternative would have been to do an implicit persist call before a query, but that seemed to be confusing.