Release 0.1.25 - Major changes to internal representation and some new examples

Overview

Summary

The focus of this release is the result of the independent intermediate
node refactoring project I was undertaking. This was to transform the list
oriented data structure for representing a "pure" intermediate form of
code into a more stuctured/generalisable format.

The motivation behind this change was the realisation that implementation
of end user functions (ie def) would be very difficult with this more
structured approach.

Rationale behind changes

Before this change, pyxie used the following data structures, and code
phases when parsing, analysing, transforming and performing code
generation.

These were not ideal:

• Code phase: Parser - read in the source, parsed it, and created a
  data structure
  • Data structure: pynodes - these represent the concrete python
    syntax, and are generated during the code generation phase. These
    are python objects in a class hierarchy. Once created they are
    analysed and placed into a context to analyse datatypes.
• Code phase: Analysis. - Analyses the code, and decorates the existing
  pynodes with type information to understand the programs data and data
  types. (does not create a new data structure).
• Transform phase: walks the pynode CST, and generates an intermediate
  data structure intended to represent the program in the abstract
  independent of the language used. An independent intermediate form
  if you like.
  • Data structure: independent intermediate form - This is used
    to model the program in a "pure form" - which isn't constrained
    by the source language, and contains enough information for the
    target language output to be generated. That was the theory. In
    practice it was a nested list of list of lists ... Not ideal.
    More on this below.
• Code generation phase: This walked the independent intermediate form
  (the lists of lists), and created an output data structure representing
  the concrete final program.
  • The third/final data structure is intended to represent the final
    output language. ie it is intended to be a concrete representation of
    the output language. This final data structure is then capable of
    creating the output. Again, forms a hierarchy and could be called
    "CppNodes" (Though they weren't before this change)

The problem here is that while the pynodes are currently well defined (to
a large extent) and that the CppNodes are pretty well defined (even if
they could be better), the independent intermediate form sucked because
it was just nested lists. This meant in practice the code was fragile,
difficult to change, and increasingly difficult to work with. In the early
days of Pyxie this simplistic structure made sense. However as data
analysis becomes more complex and tricky. This mirrors the fact that
the same was true in the early days of the parsing side.

So the focus of this sub-project was to replace the intermediate form,
and tighten up the differences and make clearer in the code base that
we have PyNodes, iiNodes, and CppNodes, where:

• Pynodes - are the current structures, unchanged - note these are
  currently prefixed Py - PyDefStatement for example.
• CppNodes - are map to objects/hierarchy/etc that represents C++ programs,
  but made clearer (slightly). These are now prefixed Cpp rather than the
  previous C_ which some of the objects use.
• iiNodes - represent the independent, intermediate nodes. Since iiNodes
  need to be used in source code where there are either PyNodes + iiNodes
  or CppNodes + iiNodes, they have the prefix "ii" enable differentiation.

Since all 3 of these are actually models, the code for these has all
moved to sit below pyxie.models.

At some point there will be a need to restructure the code more generally.
Parsing, transforming to IINodes and code generation are actually all
transforms, and should sit together. That was out of scope for this
already relatively major internal change.

Changelog

New

• doc/Versioning.md - semantic versioning as it applies to pyxie
• doc/WIPNOTES/6.Models.md - start of some docs around the models in use
• Added explicit notes on licensing of pyxie's output.
  (Short version: I view the output as being derived from your code by you)
• Language focussed examples / acceptance tests added:
  if if-else pass print while-break while-continue
• Change arduino profile to support the Adafruit_NeoPixel library
• neopixel example added

What's been fixed? / Improved

• Handling if if/elif/else improved/fixed
• added clean.sh to arduino example
• added clean.sh to servo example
• added README.md to simplest_function example (won't work yet)

Internal Changes

• bin/pyxie now pulls in functionality from pyxie.api to be clearer
  about what the API is
• added pyxie/api.py - public API for embedding pyxie. (Assumes
  ubuntu host) Core contents:
  • initialise_API(profile="default") - Call first
  • set_profile(profile_name) - Call later
  • PyxieAPI.parse(filename) - parse file, output goes to console
  • PyxieAPI.analyse(filename) - parse & analyse flle, output to console
  • PyxieAPI.codegen(filename, result_filename=None) - parse through
    code generation. Output to console
  • PyxieAPI.compile(filename, result_filename=None) - Compile file,
    result ends up as "result_filename"
• pyxie/core - changed over to drive the conversion of pynodes to
  cppnodes via iinodes. (aim is to simplify pynodes down the line and
  make iinodes our main location for analysis/etc)
• Minor changes to .gitignore
• Minor change to the Makefile to make editting simpler for me...
• Update clib.py based on changes to clib
• pyxie/codegen/simple_cpp.py - functionality shifted out to
  pyxie/models/cppnodes.py to allow a model driven approach
• pyxie/model/cppnodes.py Created. A better code representation
  model for C++. Code generation therefore means transforming from the
  iiNodes to CppNodes
• pyxie/model/iinodes.py - Introduces iiNodes - which are used to
  represent an intermediate version of a program such that it can bridge
  the conversion between languages
• pyxie/model/pynodes/operators.py - added args() method to a number
  of classes to unify behaviours
• pyxie/model/transform.py - Major change - converts PyNodes to
  iiNode representation rather than json objects.

Other

• doc/newsletter/07-20161110.Pyxie-Release-0.1.24.md corrected release date
• doc/newsletter/XX-TBD.Pyxie-2017-Goals.md - Unreleased newsletter.
  Interesting notes, will get reused if appropriate later. Rather moot
  contents now though.
• doc/newsletter/08-TBD.Focus-Or-Pyxie-Release-X.X.XX.md Template for
  next newsletter

Release 0.1.24 - Improved variable detection in else/elif, start of def statement implementation

Overview

Main focus of this release is fixing detection of variables inside
else/elif statements and internal changes.

These fixes actually enable some fun things - like the ability to control
complex robots - like robot puppies/kittens with multi-motor limbs and
sensors, so an example of doing that has been added. It's probably
the first major non-trivial example.

Additionally the ability to extend the arduino profile has been made
easier through bundling together everything to do with the arduino
profile into pyxie/profile/arduino.py . This also acts a template for
creation of other profiles. (For example, I'll add a micro:bit profile
at some point) See the changelog for more details.

There's been a number of other changes. Some internal which aren't really
user facing, but affect development - such as major reprioritisation
of the backlog (see www.sparkslabs.com/pyxie/dev-status.html). The
pyxie-dev tool I use for managing releases has been overhauled. There's
been significant improvements to the way pynodes are handled as well.

Lastly, I don't normally talk about upcoming changes, but I've started
the beginnings of implementation of user functions - ie def statements.
The python parsing side and analysis side has been completed for
functions with no arguments, no return values, and so on. However,
the conversion of this to the C++ side of things really requires
major changes to the internals of how the C++ is generated.

This will take time, so this release just contains the parsing,
and analysis side of things. The next release should contain code
generation and compilation.

Changelog

What's been fixed?

• Code generation for else/elif was failing when the statements inside
  contained identifiers. This was due to context not propogating into
  else/elif. This was caused by neither if nor elif adding else clauses
  as children. This was done and now this operates correctly.

What's New?

• Added playful puppy example that compiles and controls the Dagu Playful Puppy
  robot. (8 leg servos, 2 head/neck servos, infrared array sensor/eye)
  • v0 no funcs Playful Puppy code analyses #213
  • v0 no funcs Playful Puppy code generates code #214
  • v0 no funcs Playful Puppy code compiles, & runs on device correctly #215
• Profile specific code has been extracted to a specific file. In this
  case, the iniitial profile made more managable is of course the arduino
  profile. To configure this/extend this, you now update the file
  pyxie/profile/arduino.py
• To add more predefined variables/etc that are used in the context (ala
  A0, etc) you extend the function populate_profile_context.
• If you need to add extra types - ala the Servo type - you can use
  the Servo function call as an example. Note that it has a return
  type of Servo. This means of course that the Servo function is
  a constructor. For this to work clearly the type needs to be
  defined - so you define it below in the variable types.
• Started on parsing side aspects of definition of simple user functions.
  (ie def statements) Hopefully basic functions should be in the next
  iteration of pyxie.

Internal Changes

• pyxie-dev - Pyxie's release management tool, has had an overhaul,
  and transferred into the pyxie package, rather than standalone code.
  In the process code was improved, such that "dryrun" and "verbose"
  now mean precisely that. Help text is deliberately verbose to note
  what the release process is.
• Check all Pynodes add all sub nodes as children correctly #264
• Start of function support:
  • Lexing for function definition succeeds #265
  • Grammar parsing for function definition succeeds #266
  • Pynode support for def statements #269.
  • Add a callable type #270

Other

• Reorganised backlog based on priorities - need, useful, want, wouldlike.
  Plan going forward is to primarily focus on needs, and one or two from
  each of the other categories.
• Function call code supports simplified type definitions for externals. #34
• Block structure of generated C Code is pretty/human friendly/readable #26
• Add special case for function calls like print #37

Release 0.1.23 - major arduino profile improvements, print-as-function not statement

This release is a fairly major release - which has prompted a bump of release number to 0.1.23, rather than 0.0.23. (I'm keeping the patch level the same so there's a general release count.)

Major Changes

• The arduino profile is now sufficient for building little robots that are
  made of microcontrollers, servos, sensors and can report data via the serial
  port.
• Ability to use function calls in assignment statements - such as calls in
  to arduino libraries.
• The arduino profile is now documented - both in docs/ and also on the website.
  • http://www.sparkslabs.com/pyxie/specific-profile-arduino.html
• More examples:
  • Analog example - showing use of reading/writing analog values, and use
    of serial port, map, constrain, random, millis etc
  • puppy - this is for controls the Dagu playful puppy (a small quadruped
    robot with 10 servos and IR sensor "eye")
• print is now a function, not a statement. This was always coming, but
  this has been forced by practicalities. Things like Serial.print - necessary
  for arduino support - are not valid python if you have print as a statement.
  It's useful also that this also brings us closer to python 3's syntax.

Arduino Support

Specific set of functionality checked/added in this release:

• Serial support - .begin(), .print(), .println()
• constrain
• map
• random, randomSeed
• analogWrite
• analogRead
• millis
• Support for constants/ids : A0-A7, HIGH, LOW, INPUT, OUTPUT

This is in addition to other previous functionality like: Servos, Digital IO, time delays, etc

Changelog

What's New?

• Lots of internal changes to switch print from being a statement
  to a function call
  • Update all tests to use print as function, not as statement
  • Lexer - remove keyword
  • Disable "print" as statement type in the grammar
  • Disable "print" as a pynode in the AST model
  • Transform bare function calls to "print" (not method accesses)
    into print statements internally. This will need improving
    later, but for now is enough. It will need improving once we
    implement custom function support.
• Enable use of function calls as rvalues in assignments such as "analogRead"
• Added README.md for the analog example
• Update examples/README.md overview of the various examples.
• Initial user (single page) documentation of how to use the arduino profile

What's been fixed?

• Handle emitting identifiers as rvalues in assignments correctly

Internal changes

• Add recursive lookup through context stacks. This allows us effectively to
  manage scope, to be able to treate profiles as an enclosing scope, and a step
  towards being able to generate better error messages.
• Initial tests with profile nodes (for context). Tag is profile_identifier.
  Purpose is primarily to support analysis.
• Simplify cruftiness/verbosity of logging from parser
• Clarify source of logging messages
• Update site/src/panels/current-grammar.md to match current grammar...
  • remove print_statement
  • minor cleanups
  • Expression syntax supports expression molecules - object method access

Other

• Document how the various autogenerated docs get generated.
• Some extra scripts in test-data - designed to support quick turn around
  testing of specific language features.
• Created a default (empty) global context stack - may provide a better hook
  for things like print-as-function later on.
• Help with debugging
  • Add tags to contexts
  • Tag global context as program context
  • Tag (test) arduino profile context as "arduino"
• Support for analogRead, analogWrite, (arduino) map, and Serial object
  • Example added exercising these
• Updated docs

Pyxie 0.0.22 - Multiple Arduino Board Support (Uno/etc)

This release contains some useful, practical changes made in response to a
query I received.

To summarise the query was:

Was wondering if adding support for the Uno would be a big project?

This release provides Uno support ... and for the rest of the major arduino
boards!

The bulk of building code for the arduino is handled by the arduino
project's tools. The specific board built for is controlled by the
BOARD_TAG. This change makes it possible for you to set this for your pyxie
programs.

In particular, if you program is called my_program.pyxie , you create a file
with the name my_program.Makefile.in

(Note: If you don't create this file, you get the defaults of leonardo & /dev/ttyACM0)

In this file at present two options are supported:

• BOARD_TAG - this sets the board you're building for
• ARDUINO_PORT - this sets the specific device it's showing up as on your machine.

So, if you have an Arduino Leonardo showing up on port /dev/ttyACM0, the
my_program.Makefile.in file contains the following:

BOARD_TAG    = leonardo

(/dev/ttypACM0 is the default port!)

If you have an Dagu Mini showing up on /dev/ttypACM1, the file looks like this:

BOARD_TAG    = atmega8
ARDUINO_PORT = /dev/ttyACM1

If you have an Arduino Uno, then the file looks like this:

BOARD_TAG    = uno
ARDUINO_PORT = /dev/ttyACM1

The board tags select various options from this file:

/usr/share/arduino/hardware/arduino/boards.txt

And from that, on my system, I see the following BOARD_TAGs available:

$ grep = /usr/share/arduino/hardware/arduino/boards.txt|cut -d. -f1|uniq
uno
atmega328
diecimila
nano328
nano
mega2560
mega
leonardo
esplora
micro
mini328
mini
ethernet
fio
bt328
bt
LilyPadUSB
lilypad328
lilypad
pro5v328
pro5v
pro328
pro
atmega168
atmega8
robotControl
robotMotor

This was a relatively small change, but hopefully one that will have a neat
impact, since the above devices can now all run pyxie's
subset/incomplete/not-yet-finished flavour of python :-)

I wanted to get this out quickly though because the aim of Pyxie is to be
useful and practical. (And this struck me as particularly useful) If there
are any further requests for specific things you'd find useful sooner
rather than later, please let me know. I can't guarantee any particular
timescale - this is done on my own time after all, but the feedback is very
welcome.

Changelog

New

• Ability to override/define which arduino board you're working with.

Other

• Minor cleanups

Pyxie 0.0.21 - Arduino Servo control support

This release is a bit of a milestone. One of the things that I want to
Pyxie to be able to control is small arduino based robots. For this to
work, we need to be able to analyse sufficiently complex programs including
the ability to control servo objects. This needs the ability to call
methods on servo objects.

This release adds that ability where we wish to move things. We can attach
servos to pins, and control them. Since we can't capture results of
function calls yet, we can't detect button presses, or read analog data
values yet. So there is a way to go.

It does mean that programs like the following now parse, analyse, code
generate C++ and compile properly.

#include <Servo.h>

myservo = Servo()
pos = 0
pin=11

myservo.attach(pin)
while True:
    for pos in range(180):
        myservo.write(pos)
        delay(15)

    for pos in range(180):
        myservo.write(179-pos)
        delay(15)

For the curious, the generated code is also tidied and looks like this:

#include <Servo.h>

#include "iterators.cpp"

void setup() {
    Servo myservo;
    int pin;
    int pos;
    range range_iter_1;
    range range_iter_2;

    pos = 0;
    pin = 11;
    (myservo).attach(pin);
    while (true) {

        range_iter_1 = range(180);
        while (true) {
            pos = range_iter_1.next();
            if (range_iter_1.completed())
                break;

            (myservo).write(pos);
            delay(15);          // Itself uses pos
        }
        ;

        range_iter_2 = range(180);
        while (true) {
            pos = range_iter_2.next();
            if (range_iter_2.completed())
                break;


            (myservo).write((179 - pos));
            delay(15);          // Itself uses pos
        }
        ;
    };
}

void loop() {
}

Next step is obviously to start adding support for reading arduino sensors
(after all the focus of pyxie is on utility).

Changelog

New

• Code generation for object methods. Allows things like myservo.attach(pin) and myservo.write(pos)
• Started changes regarding how contexts will be managed/accessed by pynodes
• Global context handling improved
• Internals supported for code generation of functions extended to include attribute access

Other

• Improved notes on how context nodes will operate
• Minimal servo tests extended slightly to support moving servos

Internal

• Simplified pynode internals (removed explicit tree type)
• Cleaned up/removed extraneous code

0.0.20 The refactor release Mainly internal changes, adds WIPNOTES, updates arduino examples, prettified C++ code

This release rolls up a collection of changes made (off and on) over the past few months. The majority of them are internal changes and refactorings, but two changes may be of interest.:

• Firstly the arduino examples now include a target example for handling arduino servos in python. These are forcing a number of changes to the way type inference operates.
  • NOTE: servos do not yet analyse (and therefore don't compile) in the arduino profile, so don't be surprised if the compile/analyse scripts in that directory fail.
• Secondly, in order to for me to track better what I'm planning in various areas, what works etc, I've started creating WIPNOTES - that is NOTES designed to support current WIP. They aren't polished/etc but intended to be sufficiently up to date to be useful. Some of these may become actual docs later. Some might not.
• Thirdly. It's a detail, but I like it - the generated C++ code is now prettified. (Assuming you have /usr/bin/indent included)

The release/updates have been pushed to github, pypi and my ubuntu ppa on launchpad. (The final one will be visible whenever their systems update)

New Pyxie Release 0.0.19

The main focus of this release is the start of python 3 support and
continued work on the practicalities of creating C++ for arduinos from
python, with a primary focus on digital/analog input/output and control of
servos.

Changelog

What's New?

• Changes to support Python 3
• Now runs under python 3 as well as python 2 :-)
• (internals) Pynode docs - very much work in progress
• Arduino blink test case copied into examples
• Makes testing easier and usage more obvious

What's been fixed?

• Fix regression in code generation for function calls
• This was due to FunctionCalls now assuming they work on a callable which
  may be an identifier, but can also be other things too

Internal changes

• Massive reorg of pynodes from single file to categorised files
• Use annotated tags for releases - makes them show up on github - showing releases
• Help text for pyxie-dev extended to show release process
• Bump versions for 0.0.19

Other

• Work on arduino profile continues with more work on servos (still WIP)
• Bump copyright notices to 2016(!)
• Add link to latest release on homepage

New Pyxie Release 0.0.18

New

• Parsing of "long" integers, treating "l" as unsigned long values and "L" as signed long values.
• Code generation for long and unsigned long values works
• Add first major example of a pyxie parsable program - for controlling a 4 legged robot - DAGU playful puppy (8 servos for 4 legs, 2 for pan/tilt in head, IR Array)
  -- Does not parse/compile yet! (But will!)
• Add parsing of attribute lookup for objects
  • Parsing tests for object attribute and method access
  • Test cases for attribute access
• Add simple servo example
  • Couple of versions to simplify development - target version + simplest
    working version
• Some helper scripts to help while building/testing examples
• Redo function calls to allow attribute access ordering
• Recast identifier in functions as a callable expression
• Initial version of arduino function call descriptors
• Initial support for arduino profile in pynodes
• Code generation for first arduino specific types (specifically "Servo")
• Re-enable parse only option
• Allow expression atoms to be negatable, not just numbers (allows things like -step/2)
• Remove trailing semi-colons in arduino-blink test
• Fix Missing import regarding parsing testfiles in bin/pyxie
• Changes regarding precedence of brackets to other expression atoms

Other

• Add a list of the high level things 'missing' to Language status
• Support for dumping the parse tree results as a json file - for debug purposes (disabled in code by default)
• Restores long/unsigned hinting where necessary
• Update range to support start, end and step - replacing max
• Test case for new range implementation
• Make PyAttribute's jdump correctly
• MAJOR Clean up how options are handled - shifted into introspected classes in bin/pyxie, along with improved internal docs
• Initial cleanup inside bin/pyxie-dev
• Improve lexing error messages
• clib updated

Will need revisiting

• Do not clean up builds temporarily

New Pyxie Release 0.0.17

New

• Implemented "pass" statement.
• For loops now work on arduino profile (reimplemented C++ generators to use
  generator state, not to use an exception)
• Arduino test case using for loop

Other

• Extracted core code for "pyxie" script into pyxie.core
• Updated usage instructions to include covering using arduino profile
• Documentation updated to current status (to a large extent)

New Pyxie Release 0.0.16

Summary: Adds initial Arduino LEONARDO support, improved function call, release build scripts

In particular, to compile for a compilation target you do this:

pyxie --profile arduino some_program.pyxie

This will generate some_program.hex. You then need to load this onto your
arduino. Support for assisting with this will probably be in a largert
version. Requires Arduino.mk to be installed in a standard place. Docs TBD
as functionality stabilises.

Features

• Arduino LEONARDO compatible compilation profile (#3)
  • Detect that we are in a profile mode from a command line switch (#3.1)
  • Code generation is called with current profile (#3.2)
  • Code generation outputs code targetting "setup" instead of "main" (#3.3)
  • Makefile uses the arduino makefile #arduino leonardo (#3.4)
• Compilation profiles support removal of elements of the clib (#4.1)
• Function calls that do not require arguments work (#2)

Docs

• Docs in README.md, setup.py and docs/ are generated from website. (#66)
• Documentation in /docs is generated from website source documentation (#63)
• Docs on usage #docs (#64)
• Man page for pyxie (#65)

Other

• Make Release Script (#61)
• build_release_script.py (#62)
• Core setup.py files/etc now auto-generated from templates

Other

• Clean up build local script
• Man file added (not installed yet though)
• Build distributed docs from the same source as the website
• Added pyxie-dev script to assist in release automation
• Re-enable doc building