failing tests

[effbiae]

This is no good.

One problem is the lack of a make test. In fact, the lack of a working jconsole without having to make install is a problem, too. In future, openj requires a make test report for at least linux and windows -- ideally, we borrow a build-and-test farm from somewhere to automate the checking of all pull requests on all platforms.

Nothing but bug-fixes or reverts until we're clean.

[tavmem]

2 questions:

1. In my limited experience with Autotools, every package I've attempted to build/run has required a "make install" command. Why is that a problem in the case of onenj?

2. When the tests failed for you, it appears that you used the Autotools process. Did you try the Cmake process, and if so, did you get a working jconsole when using Linux?

[tavmem]

Just an FYI: I took a fresh clone, and did an
sudo git checkout 784c872
which took me back to to the commit done on Feb 3, 2012 at 15:15:09

I tried the cmake process (on Gentoo Linux) and diid not get a working jconsole.

I took a fresh clone again, and tried the Autotools process (on Linux).
I got a working jconsole, but got 40 failed tests.

Conclusion: The cause of the problems is earlier than the Feb 3, 2012 commit done at 15:15:09

[effbiae]

Before make install, there is conventionally a make test. This implies it should be possible to run jconsole without make install.

I settled on using cmake.

Ta, jack.

Sent from my iPhone

On 02/04/2012, at 1:24 AM, Tom [email protected] wrote:

2 questions:

1. In my limited experience with Autotools, every package I've attempted to build/run has required a "make install" command. Why is that a problem in the case of onenj?

2. When the tests failed for you, it appears that you used the Autotools process. Did you try the Cmake process, and if so, did you get a working jconsole when using Linux?

---

Reply to this email directly or view it on GitHub:
#34 (comment)

[tavmem]

Figured out why I was not getting a working jconsole when using the cmake approach:
Neglected to run the "sudo make install" step.

[mlochbaum]

Compiling on Arch Linux 64-bit, I get segfaults on g000i, g222i, g331ins, g420, and gbpar.ijs. It looks like this happens when a lot of comparisons (and possible other boolean operators) are done using rank. I'll look into the special code for =, >:, etc. when I have the chance.

I get testing failures on 23 files, but I never got a clean build with this source, and this doesn't seem like more than I had in the beginning.

We need a place to post, compare, sort, and address test results. What does Github have to accommodate this?

[effbiae]

gIthub doesn't have much more than "issues", "messages" and "wiki" as far
as i can see.

i have reports that all test cases pass with the original source release,
so, somewhere, a bug or bugs have been introduced. priority is to see
which pulls introduced the failures.

ta, jack.

[tavmem]

I tried git checkout fd62083
which was done on May 19, 2011.

Using cmake: got segfault on gstack, and exit to bash.
Using autotoools: all tests completed with testing errors on 37 files.

[effbiae]

failures on git checkout 5f49d52eb7954f24d86641cae48868d03c56a834

which is from 28 Mar 2011:

BAD ddall
/home/jack/core/test/g128x5.ijs
/home/jack/core/test/g15x.ijs
/home/jack/core/test/g320ipt.ijs
/home/jack/core/test/g410.ijs
/home/jack/core/test/g421p.ijs
/home/jack/core/test/g7x5.ijs
/home/jack/core/test/gdll.ijs
/home/jack/core/test/gibs.ijs
/home/jack/core/test/gmbx.ijs
/home/jack/core/test/gmbx0.ijs
/home/jack/core/test/gmbx1.ijs
/home/jack/core/test/gmbx2.ijs
/home/jack/core/test/gmbx3.ijs
/home/jack/core/test/gmbx4.ijs
/home/jack/core/test/gmbx5.ijs
/home/jack/core/test/gmbx6.ijs
/home/jack/core/test/gmbxah.ijs
/home/jack/core/test/gmbxip.ijs
/home/jack/core/test/gmbxqz.ijs
/home/jack/core/test/gmbxx.ijs
/home/jack/core/test/gmmf.ijs

jack@ubuntu:~/core$ git diff CMakeLists.txt
diff --git a/CMakeLists.txt b/CMakeLists.txt
index e1c276e..fd38b68 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -52,7 +52,7 @@ endif ()

add_library(j SHARED ${libj_SRCS})

set_target_properties(j PROPERTIES COMPILE_FLAGS "-fPIC -O3 -fno-strict-aliasin

+target_link_libraries(j ${jconsole_LIBS})
add_executable(jconsole ${jconsole_SRCS})
target_link_libraries(jconsole ${jconsole_LIBS})

[effbiae]

using autoreconf -i on

git checkout 34bb7c0 #Mar 23

BAD ddall
/home/jack/core/test/g128x5.ijs
/home/jack/core/test/g15x.ijs
/home/jack/core/test/g320ipt.ijs
/home/jack/core/test/g410.ijs
/home/jack/core/test/g421p.ijs
/home/jack/core/test/g7x5.ijs
/home/jack/core/test/gdll.ijs
/home/jack/core/test/gibs.ijs
/home/jack/core/test/gmbx.ijs
/home/jack/core/test/gmbx0.ijs
/home/jack/core/test/gmbx1.ijs
/home/jack/core/test/gmbx2.ijs
/home/jack/core/test/gmbx3.ijs
/home/jack/core/test/gmbx4.ijs
/home/jack/core/test/gmbx5.ijs
/home/jack/core/test/gmbx6.ijs
/home/jack/core/test/gmbxah.ijs
/home/jack/core/test/gmbxip.ijs
/home/jack/core/test/gmbxqz.ijs
/home/jack/core/test/gmbxx.ijs
/home/jack/core/test/gmmf.ijs
/home/jack/core/test/gpoly.ijs
/home/jack/core/test/gxco.ijs

[tavmem]

Just as a check that nothing changed in my environment that is affecting the tests,
I reran the tests in my build from the Jsoftware tarball (which was furher modified with suggestions by Bill Lam).
I only get failures in 2 tests:
g300t, which are timing tests.
g331ins, which are space-used tests.
Errors in these test scripts can be ignored.

[tavmem]

With git checkout b7c66ba
(which is the original load of tarball on 7 Mar 2011 at 16:08:58), I get

tom@xps /usr/local/core $ sudo touch .c
tom@xps /usr/local/core $ sudo bin/build_jconsole
cc jconsole.c -o jconsole
/tmp/ccfIAADL.o: In function main': jconsole.c:(.text+0x223): undefined reference tojepath'
jconsole.c:(.text+0x232): undefined reference to jeload' jconsole.c:(.text+0x24c): undefined reference tojefail'
jconsole.c:(.text+0x334): undefined reference to jefirst' jconsole.c:(.text+0x351): undefined reference tojedo'
collect2: ld returned 1 exit status
make: ** [jconsole] Error 1
failed: jconsole NOT created

This indicates that the full tarball was not initially loaded, but this may have been rectified in later commits.

[tavmem]

tom@xps /usr/local/core $ sudo git checkout c75fa0a
Note: checking out 'c75fa0aeba92a3f8b3a36c6228a8993d80fb0485'.

HEAD is now at c75fa0a... builds jconsole
tom@xps /usr/local/core $ sudo touch .c
tom@xps /usr/local/core $ sudo bin/build_jconsole
cc jconsole.c -o jconsole
/tmp/cc7CbZMG.o: In function main': jconsole.c:(.text+0x223): undefined reference tojepath'
jconsole.c:(.text+0x232): undefined reference to jeload' jconsole.c:(.text+0x24c): undefined reference tojefail'
jconsole.c:(.text+0x334): undefined reference to jefirst' jconsole.c:(.text+0x351): undefined reference tojedo'
collect2: ld returned 1 exit status
make: ** [jconsole] Error 1
failed: jconsole NOT created

This is the commit of 7 Mar 2011 at 9:43 which is labelled "builds jconsole"
Not clear that any of the commit states will run a full set of tests w/o errors.

[tavmem]

12 Mar 2011 08:57:59 (Clean up minor whitespace issues). Get same results:

tom@xps /usr/local/core $ sudo git checkout 5084cb3
Note: checking out '5084cb33c824d2345e04c82bcc26b9ce7c33e800'.

HEAD is now at 5084cb3... Clean up minor whitespace issues.
tom@xps /usr/local/core $ sudo touch .c
tom@xps /usr/local/core $ sudo bin/build_jconsole
cc jconsole.c -o jconsole
/tmp/ccNGEojn.o: In function main': jconsole.c:(.text+0x223): undefined reference tojepath'
jconsole.c:(.text+0x232): undefined reference to jeload' jconsole.c:(.text+0x24c): undefined reference tojefail'
jconsole.c:(.text+0x334): undefined reference to jefirst' jconsole.c:(.text+0x351): undefined reference tojedo'
collect2: ld returned 1 exit status
make: ** [jconsole] Error 1
failed: jconsole NOT created

[tavmem]

BTW:
Regarding the comment by mlochbaum:
"We need a place to post, compare, sort, and address test results. What does Github have to accommodate this?"

The "Gist" capability in github looks like it may help here.

[tavmem]

In the initial load of the tarball into github, the "makefile" was not included.

[tavmem]

There was also no j/system/defs directory setup in the initial github load.
This would cause the command bin/build_defs to fail.

[tavmem]

Summary of findings so far:

Procedure: Took checkout of the initial load of tarball to github. Then:

1. Copied in makefile.
2. Added directory j/system/defs
3. Added changes to jconfig j.h js.h and ar.c suggested by Bill Lam back in Dec 2011.

Results:

1. If I copy the resulting directory to ~ and do the builds,
   then I only get errors in scripts g300t and g331ins (timing and space-used).
2. If I copy the resulting directory to /usr/local and do the builds,
   then I get errors in 31 scripts.

[effbiae]

Nice work. Got a fix?

The missing Makefile had to do with git not liking an existing Makefile when auto tools also generates it. Maybe if I worked in a git shop, I'd appreciate it more, but I just don't like it. Should have gone with hg on google or bzr on launch pad.

But bugs sound like j can't find files where it expects them...

Aargh

Sent from my iPhone

On 04/04/2012, at 12:18 AM, Tom [email protected] wrote:

Summary of findings so far:

Procedure: Took checkout of the initial load of tarball to github. Then:

1. Copied in makefile.
2. Added directory j/system/defs
3. Added changes to jconfig j.h js.h and ar.c suggested by Bill Lam back in Dec 2011.

Results:

1. If I copy the resulting directory to ~ and do the builds,
   then I only get errors in scripts g300t and g331ins (timing and space-used).
2. If I copy the resulting directory to /usr/local and do the builds,
   then I get errors in 31 scripts.

---

Reply to this email directly or view it on GitHub:
#34 (comment)

[tavmem]

Don't have access to my Linux box till very late tonight.
Will file a fix sometime tomorrow morning.

[tavmem]

Created a branch named "jsrc" in public repository "tavmem/core".
This branch passes all j tests on my machine, which is a 32-bit Pentium-4, running Gentoo Linux.
The makefile was added and seemed to cause no problems in git.
The repository must be loaded into ~ for the tests to pass. It's not that J cannot find the files it expects.
Rather, the errors in the 31 scripts that fail if loaded into /usr/local seem to be caused by J not having permission
to write files to the directories it is targetting.
This is not a general solution. The modifications made to jconfig are tailored to getting the test scripts to
pass on my machine.

[effbiae]

Hi Tom,

Thanks so much for identifying the problem. While the failures may have an explanation, this exercise has highlighted the lack of basic quality control openj has.

A solution i'm considering is for all tests to pass on all platforms, by having a number of people, each with one or more of the platforms we support, to pull in requests and run the tests. Ideally, automated via email, but initially, we can automate sending an email to volunteers, and volunteers manually responding after they have run the tests.

Meanwhile, I'll work on make test that can be run prior to make install.

Thanks Again.

[tavmem]

Hi Jack -

Re: "The missing Makefile had to do with git not liking an existing Makefile when auto tools also generates it."

Similar problem arose in branch "jsrc" of tavmem/core with the file j/system/defs/hostdefs_linux.ijs.
Need to include a stub of the file in the commit or else the directory j/system/defs/ does not exist in the commit.
Running the builds in branch "jscrc" creates a new version of j/system/defs/hostdefs_linux.ijs
Git refuses to allow changing the branch to "master" till the new version is committed or stashed.
However, "git checkout j/system/defs/hostdefs_linux.ijs" restores the stub, and resolves the problem.

I expect a similar procedure would work to discard an Autotools generated Makefile that was not to be retained.
(But maybe I'm missing your point as to specifically what git did not like.)

Tom

[effbiae]

Tom - seeing as how you're doing such good work, can you take over the processes to run tests, and fix the install so that the tests pass before and after install? I'm having trouble finding the time - and to re-acquaint myself with core source.

No problem if you can't. Nice work with the test code and the cygwin port!

[tavmem]

Hi Jack -

I can certainly volunteer to work on these issues. Several caveats:

1. Going up steep learning curves on Autotools, C source code, and J (my background is in APL and A+).
2. Have no familiarity with Cmake, yet.
3. Have only 2 environments to test in: 32 bit Gentoo Linux, and a 64 bit laptop w Windows 7 Home Premium & Cygwin).
4. My time is somewhat spotty: (e.g.: Be away starting this morning till Monday night with limited internet access).

BTW: Still have work to do on the cygwin port:

1. The commit that I made last night works fine on cygwin, but now causes Gentoo Linux to fail (needs adjustment).
2. 26 scripts still fail in cygwin. (17 on cd domain, 6 on timing, 1 each on: socket, !y limit error, dll calls)

That being said, I'd be happy to have a go at it (or at least help out).
Tom

[lemenkov]

1. Going up steep learning curves on Autotools, C source code, and J (my background is in APL and A+).

I can help with that. Itts not so hard btw.

[tavmem]

That would be great. Thanks !!

[tavmem]

Hi Jack -

Just put in the latest commit to tavmem/core (the jsrc branch).
It implements "make test" for post install testing.

Tom

[effbiae]

Hi Tom,

Thanks for this contribution. Being able to test a build is Programming-101
and it's a bit embarrassing that openj has been alive for more than a year
without a sane test procedure.

If you are able, please continue to develop so that you can test a local
jconsole prior to install. That should mature the code just that little
bit more.

Again, Thanks.

Jack.

On Wed, Apr 11, 2012 at 4:00 AM, Tom Szczesny <
[email protected]

wrote:

Hi Jack -

Just put in the latest commit to tavmem/core (the jsrc branch).
It implements "make test" for post install testing.

Tom

---

Reply to this email directly or view it on GitHub:
#34 (comment)

[tavmem]

OK: The latest commit to branch jscr of tavmem/core.git has:
a build process for jconsole that does not install
a build process for libj that does not install
make check (that runs the test scripts on the local version of jconsole and libj)
make install (that installs jconsole and libj)
make test (that runs the test scripts on the installed versions of jconsole and libj)
(Full procedure is listed as a comment to that commit.)

Tom

[tavmem]

Hi Jack -

When running the test scripts in 32-bit Linux w/o the makefile:
load 'test/test.ijs'
bad =. TEST ddall
BAD ddall
script g331ins (space-used) always fails and g330t (timing) sometimes fails.

When using "make check" or "make test" 5 additional scripts fail (g18x gnan gpco gpci gpick).
Bill Lam suspects that this problem may be due to random data being generated for tests in the scripts,
but the same results always occur. Bill's hypothesis needs to be tested, and the problem fixed.

Tom

[effbiae]

now that is weird. it seems clear to me that there is a difference between
the binaries built by the original system and the binaries built by the
autotools system.

but try running the new jconsole and type
load 'test/test.ijs'
bad =. TEST ddall
BAD ddall

just in case jconsole test/test.ijs behaves differently... but grasping
at straws here.

ta, jack

On Wed, Apr 11, 2012 at 10:09 PM, Tom Szczesny <
[email protected]

wrote:

Hi Jack -

When running the test scripts in 32-bit Linux w/o the makefile:
load 'test/test.ijs'
bad =. TEST ddall
BAD ddall
script g331ins (space-used) always fails and g330t (timing) sometimes
fails.

When using "make check" or "make install" 5 additional scripts fail (g18x
gnan gpco gpci gpick).
Bill Lam suspects that this problem may be due to random data being
generated for tests in the scripts,
but the same results always occur. Bill's hypothesis needs to be tested,
and the problem fixed.

Tom

---

Reply to this email directly or view it on GitHub:
#34 (comment)

[tavmem]

Autotools is not building the binaries.

The procedure is:
bin/bulid_jconsole
bin/build_libj
bin/build_defs
bin/build_tsdll
make check
make install
make test

My hypothesis is that it has something to do with the nature of the shell task that is invoked.
In fact, if I take Autotools completely out of the picture, I get similar results:

If, from a cygwin bash shell I do:
~/core $ j/bin/jconsole
load 'test/test.ijs'
bad =. TEST ddall
BAD ddall
then I get 8 failures (g220t g300t g330 g420t gft gibst gipht git)

Exit from J. Then, from a cygwin bash shell do:
~/core $ j/bin/jconsole 'test/testBLD.ijs'
then I get 5 additional failures (g18x gnan gpco gpi gpick)

These are the same 5 additional failures I got in 32-bit Linux (with "make test").

Note: "make test" executes
j/bin/jconsole 'test/testBLD.ijs'
which simply runs the tests and logs the results.
(I get the same results if I take the logging out of test/testBLD.ijs)

[tavmem]

More interesting resuts:
cp test/test.ijs test/test0.ijs
In test0.ijs remove the comment indicator (NB.) from the single line:
NB. bad=: TEST ddall

If I start up J in cygwin and execute
load 'test/test.ijs'
bad =. TEST ddall
BAD ddall
I consitently get 7, 8 or 9 failures.

If I start up J and execute
load 'test0.ijs' NB. The tests will automatically run.
BAD ddall
I consistently get 12, 13 or 14 failures.

The extra 5 failures are always g18x gnan gpco gpi gpick