VGASPECx files contain the terrain for one entire level as a single giant bitmap, so it might be tricky to try to "extract" tiles out of it, but feel free to try.
The format for Lemming's tileset files are well known, so theoretically it would be possible to create a new tileset, but there are no tools right now to do so. But if we're talking Lemmix only, it would probably be easier if Lemmix supports a new file format for custom tilesets. Advantages would include no need for specialize tools to create tilesets (you could just have one or more bitmap files for the graphics, and a text file containing information mapping tile numbers to bitmap files), and ability to support 24-bit colors (ie. no longer limited to 16 colors as in Lemmings). Basically something like how Cheapo does custom tilesets (possibly even adopting the very file formats Cheapo uses for custom tilesets).