Twig

Extending Twig in as many ways as you can think of

or

Twig internals

or

Diving deep into Twig

Twig

Armin Ronacher

Developed Jinja in 2006 (Python)
Developed Twig (PHP)

Fabien Potencier

Lead developer of the Symfony project
Was looking for a Django-like templating engine
Found Twig and started "hacking" on it

Me

Matthias Noback

Dutch
Living in Zeist with my girlfriend and son
Started as freelance webdesigner (2003)
Was employed at Driebit as front-end and back-end developer
Am employed at IPPZ as application developer

Overview

Quick demo
Twig Environment
Loaders
Lexer
Parser
Token parsers
Node visitors
Compiler
Extensions
- Functions
- Filters
- Tests
- Operators
- ...

Installing Twig

In composer.json:

{
    "require": {
        "twig/twig": "1.*"
    }
}

Demo

Example: `base.html.twig`

<html>
    <body>

        <div class="sidebar">

            {{ block('sidebar') }}

        </div>

        <div class="content">

            {{ block('content') }}

        </div>

    </body>
</html>

Demo

Example: `users.html.twig`

{% extends "base.html.twig" %}

{% block sidebar %}
    <p>Nothing to see here</p>
{% endblock %}

{% block content %}

    {% for user in users %}

        <h2>{{ user.name|capitalize }}</h2>

        {% include "profile.html.twig" with { 'profile' : user.profile }

    {% else %}

        <p>No items</p>

    {% endfor %}

{% endblock %}

Demo

Example: `form.html.twig`

{% extends "base.html.twig" %}

{# use the block "sidebar" from another template #}

{% use "sidebar.html.twig" %}

{% block content %}

    {% if logged_in_user is defined %}

        <p>Hi there, {{ logged_in_user }}</p>

    {% else %}

        <p>Log in please</p>

    {% endif %}

{% endblock %}

Twig Environment

1200+ LOC
Configuration
Loading (i.e. lexing, parsing, compiling)
Caching
Rendering

$env = new \Twig_Environment();

echo $env->render('myTemplateFile.html.twig', array(
    'someVariable' => 'theValue'
);

// Twig: "uhm, what is this 'myTemplateFile.html.twig'?"

Loading a template

String loader

$stringLoader = new \Twig_Loader_String();

$env->setLoader($stringLoader);

echo $env->render('<p>Hi {{ name }}!</p>', array(
    'name' => '...'
));

Loading a template

Array loader

$arrayLoader = new \Twig_Loader_Array(array(
    'just_a_name' => '{{ name }} is in an array',
));

$env->setLoader($arrayLoader);

echo $env->render('just_a_name', array('name' => '...'));

Loading a template

Filesystem loader

$filesystemLoader = new \Twig_Loader_Filesystem(array(
    __DIR__.'/Resources/views',
    __DIR__.'/../Matthias/Demo/Resources/views',
    // ...
));

$env->setLoader($filesystemLoader);

echo $env->render('index.html.twig', array('name' => '...');

Loading a template

Chain loader

$chainLoader = new \Twig_Loader_Chain(array(
    $filesystemLoader,
    $arrayLoader,
    $stringLoader,
    // ...
));

$env = new \Twig_Environment($chainLoader);

echo $env->render('do_your_best_to_find_it');

Loading a template

Custom loader

Implement \Twig_LoaderInterface
Fetch template from your own source (like a database)

$chainLoader->addLoader(new DatabaseLoader($connection));

Or with Symfony2, create a service with the tag "twig.loader".

interface Twig_LoaderInterface
{
    public function getSource($name);

    public function isFresh($name, $time);

    public function getCacheKey($name);
}

Loading a template

class Twig_Environment
{
    // ...

    public function render($name, array $context = array())
    {
        return $this->loadTemplate($name)->render($context);
    }
}

From an array of previously loaded templates
From a cache directory (Symfony2: /app/cache/{environment}/twig)
Or
1. Retrieve the template source from the template loader
2. Compile the template source
3. Store the compiled template in the cache directory
4. Require the cached file
5. Store an instance in the array cache

Compiled template

/* index.html.twig */
class __TwigTemplate_d1d2705938bfae31fd9839ce0fe15e96 extends Twig_Template
{
    // ...

    protected function doDisplay(array $context, array $blocks = array())
    {
        // line 1
        if (isset($context["name"])) { $_name_ = $context["name"]; } else { $_name_ = null; }
        echo twig_escape_filter($this->env, $_name_, "html", null, true);
        echo " is in a file
";
    }

    public function getTemplateName()
    {
        return "index.html.twig";
    }

    // ...
}

Compiling the template

Lexer

Matches the input string against known patterns (called "lexemes")
Determines token types for these matches
Returns a stream of tokens

Compiling the template

Lexer

Compiling the template

Lexer

Take this template:

{% endif %}
<ul>
    {% for item in items %}
        <li>{{ item|capitalize }}</li>
    {% endfor %}
</ul>

"Yes, but..."

$lexer = $env->getLexer();

$template = ...;

$tokenStream = $lexer->tokenize($template);

Compiling the template

Lexer

Token stream

echo $tokenStream;

`BLOCK_START_TYPE()`	`{%`
`NAME_TYPE(endif)`
`BLOCK_END_TYPE()`	`%}`
`TEXT_TYPE(<ul>)`	raw template data
`BLOCK_START_TYPE()`	`{%`
`NAME_TYPE(for)`
`NAME_TYPE(item)`
`OPERATOR_TYPE(in)`
`NAME_TYPE(items)`
`BLOCK_END_TYPE()`	`%}`
`TEXT_TYPE(<li>)`	raw template data
`VAR_START_TYPE()`	`{{`
`NAME_TYPE(item)`
`PUNCTUATION_TYPE(\|)`
`NAME_TYPE(capitalize)`
`VAR_END_TYPE()`	`}}`
`TEXT_TYPE(</li>)`	raw template data
`BLOCK_START_TYPE()`	`{%`
`NAME_TYPE(endfor)`
`BLOCK_END_TYPE()`	`%}`
`TEXT_TYPE(</ul>)`	raw template data
`EOF_TYPE()`	end of input

Compiling the template

Lexer

Finding Twig blocks

The lexer first checks for the position of the main markers:

Start of block: {%
Start of variable: {{
Start of comment: {#

Next, the lexer

iterates over the resulting positions, while
checking some basic syntax rules, and
collecting tokens on its way to the end of the input string

Compiling the template

Lexer

Token types

Tokens have a type, a value and a line number

// types as defined in \Twig_Token
const EOF_TYPE                  = -1; // end of input
const TEXT_TYPE                 = 0;  // template text
const BLOCK_START_TYPE          = 1;  // {%
const VAR_START_TYPE            = 2;  // {{
const BLOCK_END_TYPE            = 3;  // %}
const VAR_END_TYPE              = 4;  // }}
const NAME_TYPE                 = 5;  // for, if, etc.
const NUMBER_TYPE               = 6;  // number
const STRING_TYPE               = 7;  // '...' or "..."
const OPERATOR_TYPE             = 8;  // +, -, ~, etc.
const PUNCTUATION_TYPE          = 9;  // |, [, {, etc.
const INTERPOLATION_START_TYPE  = 10; // #{
const INTERPOLATION_END_TYPE    = 11; // }

Compiling the template

Lexer

Validating the syntax

Each block and variable should be closed
```
{% for {% if
```
Brackets ({[ should be closed symmetrically
```
{{ ['a' }}
```
Closing brackets ]}) can not occur first
```
{{ ] }}
```
Unexpected characters in expressions
```
{{ \ }}
```

Compiling the template

Lexer

Validating the syntax (continued)

Unclosed comments
```
{# comment
```
Unclosed string interpolations
```
{{ "#{variable" }}
```

Compiling the template

Lexer

States

To keep track of what the lexer is doing, it uses a stack of states.

// defined in \Twig_Lexer
const STATE_DATA           = 0; // lexing data (start state)
const STATE_BLOCK          = 1; // lexing a block
const STATE_VAR            = 2; // lexing a variable
const STATE_STRING         = 3; // lexing a string
const STATE_INTERPOLATION  = 4; // lexing a string interpolation

Compiling the template

Lexer

{% endif %}
<ul>
    {% for item in items %}
        <li>{{ item|capitalize }}</li>
    {% endfor %}
</ul>

Consecutive states

`STATE_DATA`	template data
`STATE_BLOCK`	block `endif` starts
`STATE_BLOCK`	block `endif` ends
`STATE_DATA`	`<ul>`
`STATE_BLOCK`	block `for` starts
`STATE_BLOCK`	name: `item`
`STATE_BLOCK`	name: `in`
`STATE_BLOCK`	name: `items`
`STATE_BLOCK`	block `for` ends
`STATE_DATA`	`<li>`
`STATE_VAR`	variable starts, name: `item`
`STATE_VAR`	punctuation: `\|`
`STATE_VAR`	name: `capitalize`
`STATE_VAR`	variable ends
`STATE_DATA`	`</li>`
`STATE_BLOCK`	block `endfor` starts
`STATE_BLOCK`	block `endfor` ends
`STATE_DATA`	`</ul>`

Compiling the template

Lexer

States on a stack

Push and pop: add one on top, take one from the top (last in, first out)

- STATE_DATA
- STATE_BLOCK
- STATE_DATA
- STATE_DATA
- STATE_BLOCK
- STATE_DATA
- STATE_DATA
- STATE_VAR
- STATE_DATA
- STATE_DATA
- STATE_BLOCK
- STATE_DATA
- STATE_DATA
[EOF]

Compiling the template

Lexer

The resulting list of tokens may be semantically incorrect,
i.e. its meaning may be unclear or it may have no meaning at all.

In the Twig language, that is...

{% endif %}
<ul>
    {% for item in items %}
        <li>{{ item|capitalize }}</li>
    {% endfor %}
</ul>

`BLOCK_START_TYPE()`	`{%`
`NAME_TYPE(endif)`
`BLOCK_END_TYPE()`	`%}`
...

Parsing the token stream

The parser will

Process the token stream and en passant
build an Abstract Syntax Tree for the template.

Tokens will be

discarded, or
replaced by nodes

Nodes describe unequivocally the desired semantics of the original input string.

Parsing the token stream

Creating the Abstract Syntax Tree

<ul>
    {% for item in items %}
        <li>{{ item|capitalize }}</li>
    {% endfor %}
</ul>

"Hey, where is the..."

$template = ...;

$tokenStream = $lexer->tokenize($template);

$parser = $env->getParser();

$nodeTree = $parser->parse($tokenStream);

echo $nodeTree;

Parsing the token stream

Excerpt of the Abstract Syntax Tree

Twig_Node_Module(
  body: Twig_Node_Body(
    0: Twig_Node(
      0: Twig_Node_Text(data: '<ul>    ')
      1: Twig_Node(
        0: Twig_Node_SetTemp(name: 'items')
        1: Twig_Node_For(
          value_target: Twig_Node_Expression_AssignName(name: 'item')
          seq: Twig_Node_Expression_TempName(name: 'items')
          body: Twig_Node(
            0: Twig_Node(
              0: Twig_Node_Text(data: '        <li>')
              1: Twig_Node(
                0: Twig_Node_SetTemp(name: 'item')
                1: Twig_Node_Print(
                  expr: Twig_Node_Expression_Filter(
                    node: Twig_Node_Expression_Filter(
                      node: Twig_Node_Expression_TempName(name: 'item')
                      filter: Twig_Node_Expression_Constant(value: 'capitalize')
                    )
                    filter: Twig_Node_Expression_Constant(value: 'escape')
                  )
...

Parsing the token stream

The module node

The parsing process results in a \Twig_Node_Module containing

the body of the template,
or a collection of blocks,
the link to a parent template,
macros,
traits,
and embedded templates

Parsing the token stream

The main token types

The \Twig_Parser::parse() method collects nodes based on the token at the current position in the token stream.

`TEXT_TYPE`	template text	create a `\Twig_Node_Text` with the value of the token
`VAR_START_TYPE`	variable	parse the expression that follows and expect a `VAR_END_TYPE`
`BLOCK_START_TYPE`	block with a tag	expect a name, which is the name of the tag (i.e. for, if, etc.) and call a subparser

Parsing the token stream

A subparser is also known as a "token parser"

Bad name though...

It allows you to define tags including any syntax, eventually followed by a BLOCK_END_TYPE token.

{% for item in items %}
   ...
{% endfor %}

Parsing the token stream

Returning nodes

The parse() method of a token parser should return null or an instance of \Twig_Node

Returned nodes are inserted in the Abstract Syntax Tree

Twig contains already many token parsers and nodes for tags like

block
for
if
set
extends
...

You can easily override the behavior of these tags.

Parsing the token stream

A custom token parser

class UsergroupTokenParser extends \Twig_TokenParser
{
    public function parse(\Twig_Token $token)
    {
        // immediately expect "end of block", no arguments
        $this->parser->getStream()->expect(\Twig_Token::BLOCK_END_TYPE);

        $expr = new \Twig_Node_Expression_Constant('AmsterdamPHP',
            $token->getLine());

        return new \Twig_Node_Print($expr, $token->getLine(),
            $this->getTag());
    }

    public function getTag()
    {
        return 'usergroup';
    }
}

{% usergroup %}

Parsing the token stream

Using the custom token parser

There we go, {% usergroup %}!

$template = ...;

$env = new \Twig_Environment($loader);

$env->addTokenParser(new UsergroupTokenParser());

echo $env->parse($env->tokenize($template));

Parsing the token stream

Excerpt of the Abstract Syntax Tree

Twig_Node_Module(
  body: Twig_Node_Body(
    0: Twig_Node(
      0: Twig_Node_Text(data: 'There we go, ')
      1: Twig_Node_Print(
        expr: Twig_Node_Expression_Constant(value: 'AmsterdamPHP')
      )
      2: Twig_Node_Text(data: '!')
    )
  )
)

Parsing the token stream

Expressions

{{ 5 + age * 4 }}

$template = ...;

$moduleNode = $env->parse($env->tokenize($template));

echo $moduleNode;

Expressions are parsed by a specialized expression parser.

Parsing the token stream

Expressions

Twig_Node_Module(
  body: Twig_Node_Body(
    0: Twig_Node(
      0: Twig_Node_SetTemp(name: 'age')
      1: Twig_Node_Print(
        expr: Twig_Node_Expression_Filter(
          node: Twig_Node_Expression_Binary_Add(
            left: Twig_Node_Expression_Constant(value: 5)
            right: Twig_Node_Expression_Binary_Mul(
              left: Twig_Node_Expression_TempName(name: 'age')
              right: Twig_Node_Expression_Constant(value: 4)
            )
          )
          filter: Twig_Node_Expression_Constant(value: 'escape')
          arguments: Twig_Node(
            0: Twig_Node_Expression_Constant(value: 'html')
          )
        )
      )
    )
  )
)

Parsing the token stream

Expressions

Associativity and precedence

Most operators are left associative, which means a + b + c is to be read as

((a + b) + c)

and not as

(a + (b + c))

Operators have a number indicating their precedence, so a + b * c will always be interpreted as

(a + (b * c))

since the "*" operator has a higher precedence then the "+" operator.

Parsing the token stream

Node visitors

Just after all tokens are parsed and all the nodes are created node visitors are allowed to revisit the entire node tree and modify, add, replace or remove nodes. This allows for awesome and also dangerous things to happen.

interface Twig_NodeVisitorInterface
{
    // called before child nodes are visited.
    public function enterNode(Twig_NodeInterface $node, Twig_Environment $env);

    // called after child nodes are visited.
    public function leaveNode(Twig_NodeInterface $node, Twig_Environment $env);

    // returns the priority for this visitor.
    public function getPriority();
}

See for an example: Collecting Data Across Templates Using a Node Visitor

Compiling the template

The root node of the tree of nodes is a \Twig_Node_Module which contains all the nodes generated by

the parser,
the expression parser,
the token parsers and
the node visitors.

The compiler just calls the compile() method of the resulting \Twig_Node_Module node.

{% for item in items %}
    {{ item|capitalize }}<br />
{% endfor %}

$template = ...;

echo $env->compileSource($template);

Compiling the template

Excerpt of the compiled template

class __TwigTemplate_d41d8cd98f00b204e9800998ecf8427e extends Twig_Template
{
    protected function doDisplay(array $context, array $blocks = array())
    {
        // line 1
        if (isset($context["items"])) { $_items_ = $context["items"]; } else { $_items_ = null; }
        $context['_parent'] = (array) $context;
        $context['_seq'] = twig_ensure_traversable($_items_);
        foreach ($context['_seq'] as $context["_key"] => $context["item"]) {
            // line 2
            echo "    ";
            if (isset($context["item"])) { $_item_ = $context["item"]; } else { $_item_ = null; }
            echo twig_escape_filter($this->env, twig_capitalize_string_filter($this->env, $_item_), "html", null, true);
            echo "<br />
";
        }
        $_parent = $context['_parent'];
        unset($context['_seq'], $context['_iterated'], $context['_key'], $context['item'], $context['_parent'], $context['loop']);
        $context = array_merge($_parent, array_intersect_key($context, $_parent));
    }
}

Compiling the template

UsergroupTokenParser revisited

class UsergroupTokenParser extends \Twig_TokenParser
{
    public function parse(Twig_Token $token)
    {
        // create an instance of \Twig_Node_Expression
        $expr  = ...

        return new \Twig_Node_Print(
            $expr,
            $token->getLine(),
            $this->getTag()
        );
    }
}

Compiling the template

Compiling a print node

class Twig_Node_Print
{
    public function compile(Twig_Compiler $compiler)
    {
        $compiler
            ->addDebugInfo($this)
            ->write('echo ')
            ->subcompile($this->getNode('expr'))
            ->raw(";\n")
        ;
    }
}

The result can be found in the compiled template:

protected function doDisplay(array $context, array $blocks = array())
{
    // line 1
    echo "There we go, ";
    echo "AmsterdamPHP"; // the expression was just a constant
    echo "!";
}

Compiling the template

Two important conclusions:

We can put any PHP code we want inside a template
We can do heavy calculations at compile time (just once)

We only have to create our own node type and implement the compile method.

Writing parsers and nodes can be quite difficult, so

read the compiled templates in your cache directory and
write unit tests for your custom parser and node type (extend from \Twig_Test_NodeTestCase)

Extending Twig

There are some very "deep" ways to modify Twig's behavior:

$env->setLoader($loader): for instance load templates from a database
$env->setLexer($lexer): could be a bad idea (when you want to change basic syntax, use options)
$env->setParser($parser): probably a bad idea

$options = array(
    'tag_comment'     => array('{#', '#}'),
    'tag_block'       => array('{%', '%}'),
    'tag_variable'    => array('{{', '}}'),
    'whitespace_trim' => '-',
    'interpolation'   => array('#{', '}')
);

$lexer = new \Twig_Lexer($env, $options);

$env->setLexer($lexer);

Extending Twig

Extensions

interface Twig_ExtensionInterface
{
    public function initRuntime(Twig_Environment $environment);

    public function getTokenParsers();

    public function getNodeVisitors();

    public function getFilters();

    public function getTests();

    public function getFunctions();

    public function getOperators();

    public function getGlobals();

    public function getName();
}

$env->addExtension($extension) // is the way

Or when using Symfony2: create a service with a tag "twig.extension"

Extending Twig

Functions

class MyExtension extends \Twig_Extension
{
    public function getFunctions()
    {
        return array(
            new \Twig_SimpleFunction('myFunction',
                array($this, 'myFunctionMethod'),
                array(
                    // 'needs_environment' => true,
                    // 'needs_context' => true,
                    'is_safe' => array('html')
                )
            )
        );
    }

    public function myFunctionMethod($thing)
    {
        return sprintf('This is <b>my</b> %s.', $thing);
    }
}

Extending Twig

Functions

echo $env->render('{{ myFunction("<strong>computer</strong>") }}');

Will render...

This is <b>my</b> <strong>computer</strong>.

Extending Twig

Filters

class MyExtension extends \Twig_Extension
{
    public function getFilters()
    {
        return array(
            new \Twig_SimpleFilter('mine',
                array($this, 'myFilterMethod'),
                array(
                    // 'needs_environment' => true,
                    // 'needs_context' => true,
                    'is_safe' => array('html'),
                    'pre_escape' => 'html'
                )
            )
        );
    }

    public function myFilterMethod($what, $mine = true)
    {
        return sprintf('%s (which %s mine)', $what, $mine ? 'is':'is not');
    }
}

Extending Twig

Filters

echo $env->render('{{ thing|mine(false) }}', array(
    'thing' => '<strong>Life</strong>'
));

Will render...

&lt;strong&gt;Life&lt;/strong&gt; (which is not mine)

Extending Twig

Dynamic filters

public function getFilters()
{
    return array(
        new \Twig_SimpleFilter('*_of_*', function($type, $owner, $what) {
            return sprintf('%s is a %s of %s', $what, $type, $owner);
        })
    );
}

Extending Twig

Dynamic filters

echo $env->render('{{ "This game"|call_of_duty() }}');

Will render...

This game is a call of duty

Extending Twig

Tests

class MyExtension extends \Twig_Extension
{
    public function getTests()
    {
        return array(
            new \Twig_SimpleTest('usergroup', function($name) {
                return $name === 'AmsterdamPHP';
            })
        );
    }
}

Extending Twig

Tests

echo $env->render('{% if "AmsterdamPHP" is usergroup %}I told you so{% endif %}');

Will render...

I told you so

Extending Twig

Operators

class MyExtension extends \Twig_Extension
{
    public function getOperators()
    {
        return array(
            array(
                'no' => array(
                    'precedence' => 50,
                    'class' => 'Twig_Node_Expression_Unary_Not'
                )
            ),
            array(
                'maybe' => array(
                    'precedence' => 10,
                    'class' => 'Twig_Node_Expression_Binary_Or',
                    'associativity' => \Twig_ExpressionParser::OPERATOR_LEFT)
                )
            )
        );
    }
}

Extending Twig

Operators

echo $env->render('{% if no true maybe this %}It is true{% endif %}', array(
    'this' => true
));

Will render...

It is true

Extending Twig

Token parsers and node visitors

class MyExtension extends \Twig_Extension
{
    public function getTokenParsers()
    {
        return array(
            new UserGroupTokenParser()
        );
    }

    public function getNodeVisitors()
    {
        return array(
            new UnknownNodeVisitor()
        );
    }
}

And...

Twig

Extending Twig in as many ways as you can think of

Twig internals

Diving deep into Twig

Twig

Armin Ronacher

Fabien Potencier

Me

Matthias Noback

Overview

Installing Twig

Demo

Example: base.html.twig

Demo

Example: users.html.twig

Demo

Example: form.html.twig

Twig Environment

Loading a template

String loader

Loading a template

Array loader

Loading a template

Filesystem loader

Loading a template

Chain loader

Loading a template

Custom loader

Loading a template

Compiled template

Compiling the template

Lexer

Compiling the template

Lexer

Compiling the template

Lexer

Compiling the template

Lexer

Token stream

Compiling the template

Lexer

Finding Twig blocks

Compiling the template

Lexer

Token types

Compiling the template

Lexer

Validating the syntax

Compiling the template

Lexer

Validating the syntax (continued)

Compiling the template

Lexer

States

Compiling the template

Lexer

Consecutive states

Compiling the template

Lexer

States on a stack

Compiling the template

Lexer

Parsing the token stream

Parsing the token stream

Creating the Abstract Syntax Tree

Parsing the token stream

Excerpt of the Abstract Syntax Tree

Parsing the token stream

The module node

Parsing the token stream

The main token types

Parsing the token stream

A subparser is also known as a "token parser"

Parsing the token stream

Returning nodes

Parsing the token stream

A custom token parser

Parsing the token stream

Using the custom token parser

Parsing the token stream

Example: `base.html.twig`

Example: `users.html.twig`

Example: `form.html.twig`

Thank you and
good bye