Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

open up libkfio.o #82

Open
plappermaul opened this issue Apr 25, 2021 · 26 comments
Open

open up libkfio.o #82

plappermaul opened this issue Apr 25, 2021 · 26 comments
Labels
enhancement

Comments

@plappermaul
Copy link
Collaborator

After digging around in the driver I think it might be possible to open up the precompiled libkfio.o. That would require some code rearrangement. If you think that is worth a try ... here are some first bits:

struct fusion_nand_device {               // size unknown
  char unknown_0[16];
  char bus_name[8];                       // offset  16 size unknown
  char unknown_24[248];
  char dev_name[8];                       // offset 272 size unknown
  char unknown_280[32];
  char *pci_dev;                          // offset 312  
  char unknown_320[28];
  struct miscdevice *miscdevice;          // offset 348
  char unknown_356[112];  
  uint32_t devnum;                        // offset 468
  char unknown_476[36];
  fusion_spinlock_t lock;                 // offset 512 size 192
  kfio_poll_struct *poll_struct;          // offset 704
  char unknown_712[196];
  int32_t event;                          // offset 908  
}

uint32_t fusion_nand_get_devnum(struct fusion_nand_device *nand_dev)
{
  return nand_dev->devnum;
}

miscdevice *fusion_nand_get_miscdev(struct fusion_nand_device *nand_dev)
{
  return nand_dev->miscdevice;
}
@snuf
Copy link
Collaborator

snuf commented Apr 26, 2021

Nice! I have part of it deconstructed, which I should probably check in into github. There's also parts that are not easy to deconstruct, as there are some anonymous functions, seems like these are only called from the code in the library.

@plappermaul
Copy link
Collaborator Author

Good idea.

Here are the efforts so far (at least what I was able to duplicate)

https://www.codepile.net/pile/V8gwD784
https://www.codepile.net/pile/kAd9GlgB

@plappermaul
Copy link
Collaborator Author

More than that cannot be taken over without problems.

Thats why I'm currently playing around with objedump symbol renaming :)

@snuf
Copy link
Collaborator

snuf commented Apr 26, 2021 •
edited
Loading

@plappermaul make your life easy, use Ghidra. you can choose to load the entire module in or the .so. It will decompile it and make reasonable nice C code for you ;) (that's what I did, I can't upload it now, but could do later). The C code looks very usable!

@plappermaul
Copy link
Collaborator Author

Thanks for the info.

Mostly I cared about the structures that are still hidden inside the lob. So I looked at the functions to give this meaningful names.

I have several symbols in the lob renamed and globalized (at least in the cc73 file. Should I send a new version in?

@snuf
Copy link
Collaborator

snuf commented Apr 27, 2021

@plappermaul yep please send it in if it works and makes things clearer that is definitely a good thing! The cc74 is a copy of cc63, which is the last OG compiled .so delivered with the driver. During make we just copy the last .so to the next based on the gcc version.

I've created a libkfiio branch and uploaded the .c and .h file there. It's a whopping 4.2MB, I think a bunch of stuff got decompiled in a not so good way (strings/errors and some of the graph/path handling). I think it'd be good if we can split out the external interface bits first perhaps, and then move to unravel the remains? The external interface bits should be easy to clean also I think as they're known from our side.

I also have a list of what the .so imports from the source driver as part of the symbol tree, not that that is incredibly interesting, but may be useful? I'm still figuring out how to save it though.

Don't hesitate to join the Discord ;)

@plappermaul
Copy link
Collaborator Author

pushed a commit with some initial renames. Can you have a look if everything sitll compiles nicely?

@snuf
Copy link
Collaborator

snuf commented Apr 27, 2021

Thanks!

I changed KFIO_EXPORT_SYMBOL(fusion_info_root); to KFIO_EXPORT_SYMBOL(g_fusion_info_root); to get it to compile and pushed it.

Test suite is running now :)

This was the error btw:

    default:   CC [M]  /home/vagrant/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/module_param.o
    default: In file included from ./include/linux/linkage.h:7,
    default:                  from ./include/linux/kernel.h:7,
    default:                  from ./include/asm-generic/bug.h:20,
    default:                  from ./arch/x86/include/asm/bug.h:93,
    default:                  from ./include/linux/bug.h:5,
    default:                  from ./include/linux/mmdebug.h:5,
    default:                  from ./include/linux/gfp.h:5,
    default:                  from ./include/linux/slab.h:15,
    default:                  from /home/vagrant/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/include/fio/port/linux/kfio_config.h:39,
    default:                  from /home/vagrant/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/include/fio/port/kfio_config.h:38,
    default:                  from /home/vagrant/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/port-internal.h:62,
    default:                  from /home/vagrant/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kinfo.c:29:
    default: /home/vagrant/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kinfo.c:546:20: error: ‘fusion_info_root’ undeclared here (not in a function); did you mean ‘g_fusion_info_root’?
    default:   546 | KFIO_EXPORT_SYMBOL(fusion_info_root);
    default:       |                    ^~~~~~~~~~~~~~~~
    default: ./include/linux/export.h:98:16: note: in definition of macro ‘___EXPORT_SYMBOL’
    default:    98 |  extern typeof(sym) sym;       \
    default:       |                ^~~
    default: ./include/linux/export.h:155:34: note: in expansion of macro ‘__EXPORT_SYMBOL’
    default:   155 | #define _EXPORT_SYMBOL(sym, sec) __EXPORT_SYMBOL(sym, sec, "")
    default:       |                                  ^~~~~~~~~~~~~~~
    default: ./include/linux/export.h:158:29: note: in expansion of macro ‘_EXPORT_SYMBOL’
    default:   158 | #define EXPORT_SYMBOL(sym)  _EXPORT_SYMBOL(sym, "")
    default:       |                             ^~~~~~~~~~~~~~
    default: /home/vagrant/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/include/fio/port/common-linux/kfio.h:146:33: note: in expansion of macro ‘EXPORT_SYMBOL’
    default:   146 | #define KFIO_EXPORT_SYMBOL(sym) EXPORT_SYMBOL(sym)
    default:       |                                 ^~~~~~~~~~~~~
    default: /home/vagrant/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kinfo.c:546:1: note: in expansion of macro ‘KFIO_EXPORT_SYMBOL’
    default:   546 | KFIO_EXPORT_SYMBOL(fusion_info_root);
    default:       | ^~~~~~~~~~~~~~~~~~
    default: make[2]: *** [scripts/Makefile.build:279: /home/vagrant/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/kinfo.o] Error 1
    default: make[2]: *** Waiting for unfinished jobs....
    default: make[1]: Leaving directory '/usr/src/linux-headers-5.11.10'
    default: make[1]: *** [Makefile:1808: /home/vagrant/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16] Error 2
    default: make: *** [Makefile:134: modules] Error 2
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

@plappermaul
Copy link
Collaborator Author

@snuf : Do me a favour and update the disassembly

@snuf
Copy link
Collaborator

snuf commented Apr 28, 2021

@plappermaul done, shows nice functions now, I left the cc93 in (which is a copy of the original) and did the cc74 decompile.

@plappermaul
Copy link
Collaborator Author

I just tried to add a new c library to the project but I'm missing the hook to integrate it. What do I have to do to get it compiled?

@plappermaul
Copy link
Collaborator Author

found it ... KBuild file

@snuf
Copy link
Collaborator

snuf commented May 5, 2021 •
edited
Loading

Haha, I hadn't noticed but the pci_probe you introduced and the device opening fix the enumeration of the device. Before having two devices, vsl and vsl4, the driver would conflict on the fct and fioX. After manually loading the driver on the test box I noticed this:

[  107.648552] <6>fioinf ioDrive 0000:00:04.0: Found device fct1 (Fusion-io ioDrive2 1.205TB 0000:00:04.0) on pipeline 0
[  109.403206] <6>fioinf Fusion-io ioDrive2 1.205TB 0000:00:04.0: probed fct1
[  109.483578] <6>fioinf Fusion-io ioDrive2 1.205TB 0000:00:04.0: sector_size=512
[  109.483581] <6>fioinf Fusion-io ioDrive2 1.205TB 0000:00:04.0: setting channel range data to [2 .. 2047]
[  109.814983] <6>fioinf Fusion-io ioDrive2 1.205TB 0000:00:04.0: Found metadata in EBs 245-245, loading...
[  110.161428] <6>fioinf Fusion-io ioDrive2 1.205TB 0000:00:04.0: setting recovered append point 245+749518848
[  110.169701] <6>fioinf Fusion-io ioDrive2 1.205TB 0000:00:04.0: Creating device of size 1205000000000 bytes with 2353515625 sectors of 512 bytes (515361258 mapped).
[  110.177427] fioinf Fusion-io ioDrive2 1.205TB 0000:00:04.0: Creating block device fiob: major: 251 minor: 16 sector size: 512...
[  110.177891]  fiob: fiob1
[  110.178444] <6>fioinf Fusion-io ioDrive2 1.205TB 0000:00:04.0: Attach succeeded.

The test scripts use an expression that just matches on fio when looking so I never noticed. This could mean that we can have the drivers side by side in one machine, as long as vsl4 loads first (or we backport the same fixes later to vsl4, then it doesn't matter) :)

Nice!

@plappermaul
Copy link
Collaborator Author

Thanks for the positive feedback. Nevertheless I will fix the increment

rc = iodrive_pci_attach(pci_dev, gv_init_pci_counter++);

@Tourman36
Copy link
Collaborator

Tourman36 commented May 6, 2021 via email

@snuf
Copy link
Collaborator

snuf commented May 6, 2021

@Tourman36 yes and no, it brings us closer, but there is a lot more work to be done before that would be the case.

@plappermaul
Copy link
Collaborator Author

@snuf I recovered some of the function names in the blob. Maybe you can merge them into master. Best would be to integrate them into the build scripts so we always have a readable library version at hand.

@snuf
Copy link
Collaborator

snuf commented May 26, 2021 via email

@plappermaul
Copy link
Collaborator Author

No need to hurry. We have basically two choices:

Either replace all the symbol names in the libraries once. Keep the modified libraries from then on

Or keep the libraries as is and keep a script that switches the symbol names on demand. Maybe dynamically in the build process.

Choose whatever you like.

@plappermaul
Copy link
Collaborator Author

This is what we need to do for each symbol.

objcopy --redefine-sym ifio_58422......fef60.3.2.16.1731=fio_device_get_write_retard_reason x86_64_cc53_libkfio.o_converted

@plappermaul
Copy link
Collaborator Author

All required info is in https://github.com/snuf/iomemory-vsl/blob/b2e760d384a5a14392527e1460af7699ab599297/root/usr/src/iomemory-vsl-3.2.16/kfio/function_hashes.txt

@snuf
Copy link
Collaborator

snuf commented Jun 7, 2021

Wrote some shell to do this full rename in one go based on the file provided, though the time it takes to do the obj replacement I think it would be better to do the updates to the file and check it in, instead of doing it compile time. The compile of the driver is faster than the obj renaming now, by a lot. We can incrementally update the .o I would think over releases etc, sounds like a plan?

@plappermaul
Copy link
Collaborator Author

Good idea. As the sha1 symbol names are used nowhere else feel free to permamently decode them.

@snuf
Copy link
Collaborator

snuf commented Aug 27, 2021 •
edited
Loading

Sorry for the delay on that I had to look back at where we were with that.

I have a script that does the translation however if I do it on the "vanilla" module, not the one that's bundled in the libkfio branch, I get some errors on objects that are undefined. Could it be there are some missing from the translation table you've renamed before, and if so are you able to add those to the hashes file?

# Replaced 1379 objects in x86_64_latest_libkfio.o_shipped from 3531
#
# ERROR: modpost: "g_fusion_info_root" [/home/snuf/Documents/Sources/fusion/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko] undefined!
# ERROR: modpost: "gv_lock_indirect_read" [/home/snuf/Documents/Sources/fusion/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko] undefined!
# ERROR: modpost: "g_config_table_count" [/home/snuf/Documents/Sources/fusion/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko] undefined!
# ERROR: modpost: "gv_init_pci_counter" [/home/snuf/Documents/Sources/fusion/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko] undefined!
# ERROR: modpost: "g_config_table" [/home/snuf/Documents/Sources/fusion/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko] undefined!
# ERROR: modpost: "iodrive_pci_attach_post" [/home/snuf/Documents/Sources/fusion/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko] undefined!
# ERROR: modpost: "gv_fio_device_list" [/home/snuf/Documents/Sources/fusion/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko] undefined!
# ERROR: modpost: "g_fusion_info_iodrive" [/home/snuf/Documents/Sources/fusion/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko] undefined!
# ERROR: modpost: "iodrive_pci_attach_setup" [/home/snuf/Documents/Sources/fusion/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko] undefined!
# ERROR: modpost: "gv_fusion_info_fio" [/home/snuf/Documents/Sources/fusion/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16/iomemory-vsl.ko] undefined!

I forgot if I ran tests with it, am doing that now, and if all is good I'll commit the script in the libkfio branch for "posterity". The thing however is that right now I've setup the make script to do a pass every time it's run, thought I think that is a bad idea in hindsight and we should just commit the patched binary (as it saves a lot of time when building/renaming/etc). What are your thoughts on this?

@snuf
Copy link
Collaborator

snuf commented Aug 27, 2021

All looks good btw, so PR and merge it I would say! :) (or you want me to do it?)

@plappermaul
Copy link
Collaborator Author

As long as you simply replace the decoded symbols and do not take any other of my crap from this branch. Feel free.

I'm currently busy on other projects.

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@snuf @plappermaul @Tourman36