I am in a stupid position. I think it will take me two hours with a wide error bar to figure out what is going on with the war file and starting jetty. I've spent some hours on it already without much success. So high risk. It will take me at most two hours to use my tiny web framework, write my own routing module (by hand) and then we can avoid maven, jetty, war files, etc. I'll be using the generated controller skeletons and the generated json/java objects (a huge plus) but otherwise just hand coding the routing of request URL to management method. But there are pretty much no risk, I know exactly what to do.
So my inclination is to do it myself.
jetty starts up just fine, it's only complaining the dependencies are not self contained in the .war
i think it's just a matter of declaring something in the correct list
as in your code tries to use something which is not bundled in
right. I'm sure the fix is small, but that doesn't mean I'll be able to figure it out in a reasonable time.
Also, the introspection is weird.
as maven puts that .war together, sorta needs to be in the target declaration in there
in the end so long as the rosetta folks are happy with the performance, it should be ok
any idea what sorts of traffic densities they plan to slam on one of those?
I imagine, not much. And the advantage of docker is they can just scale them up as needed
and i think it's really something this silly, like telling it to include all the jars it pulled in for building https://stackoverflow.com/a/54340181/1214697 I've searched on internet for quite some time and I'm unable to figure out how to configure the maven-war plugin or something alike so that the system dependencies are included in the built-war (WE...
What I have working performs fine with the "mvn jetty:run"
but then you have to pull maven into the runtime container, which breaks the rosetta constraint of "no build tools at runtime"
agreed
https://coderanch.com/t/594897/ide/include-maven-dependencies-war-file > Just ensure that your POM is set up to generate a WAR (and not something such as JAR, instead). > Dependencies in "compile" scope will automatically be copied to the target's WEB-INF/lib as part of the Maven build. >Dependencies in "provided" scope will not (you would use this for server-supplied jars such as the JSP-api.jar).
that seems relevant, but i am just pulling random stuff in from trivial googling here
i'd start from changing the scope of the two dependencies from compile to runtime, compile is the default scope, try adding a `<scope>runtime</scope>` to both `io.swagger` and `com.google.guava`
or do you have to declare both twice, once as compile and once as runtime?
that's what the docs seem to do there in the examples
If the problem were jetty starting with the war and then getting class not found, that would make sense.
At least what I saw was just not being able to find any of the methods which makes me think it was something else
but I should look again
it's failing to find `io.swagger`
so i guess ``` <dependency> <groupId>io.swagger</groupId> <artifactId>swagger-inflector</artifactId> <version>${swagger-inflector-version}</version> </dependency> <dependency> <groupId>io.swagger</groupId> <artifactId>swagger-inflector</artifactId> <version>${swagger-inflector-version}</version> <scope>runtime</scope> </dependency>``` is worth trying out
sure
verified, log starts off as ```loading inflector config from inflector.yaml 19:19:16.857 [main] WARN i.s.o.inflector.config.Configuration - couldn't read inflector config from system property 19:19:16.859 [main] WARN i.s.o.inflector.config.Configuration - couldn't read inflector config from system property 19:19:17.006 [main] WARN i.s.o.i.utils.ReflectionUtils - Failed to find class [io.swagger.oas.sample.controllers.Network] 19:19:17.006 [main] WARN i.s.o.i.utils.ReflectionUtils - Failed to find class [io.swagger.oas.sample.controllers.NetworkController]```
I see I need to go deep into this nonsense: https://stackoverflow.com/questions/34063694/fixed-length-64-bytes-ec-p-256-signature-with-jce I need a fixed length 64 Byte ECDSA signature with the NIST P-256 Curve. The implementation hast to use JCE. The following code sample can generate a signature and verify it. Provider provSign =...
rosetta test cli is giving my 64-byte raw signatures but I am using 70-72 byte der encoded signatures
is that just an encapsulation problem?
and i can see why they could want a standard size
yeah, if I had understood this at the time I might have gone for the raw
but either way, I just need to convert it
as you can read it, you'd have the bytes there to dump from?
yeah, I have a der parser that prints out each layer somewhere. I'll run that on my regular snowblossom signatures and figure out how to construct the DER from raw.
I had to dig into these a bit for something else a while ago
but the rosetta api is pretty fancy, the private keys don't ever touch the api server
I was initially expecting it to be more like what http://qtrade.io wanted, which was an offline server that does the signing and holds the keys
```INFO: Key report standard Pub size: 33, sig 71 INFO: Key report: 30450220181567f4ca9c0ef1fe24a788e7f9d1487bf620b5d60d44b12d228aa84827dae3022100cc81f7e25488d1ccf1e9cab36983874cd2f8c4942a95b2e5c3a5fa5a436e3cb0 INFO: Key report standard Pub size: 33, sig 71 INFO: Key report: 3045022072f15bb4d962a64faa4523215dae71507ab5d4ba48cfb7c587484e318e50021b02210080acc789532abef470360dc4b8df7072418a87fd75bcdc492df994ead80327ce INFO: Key report standard Pub size: 33, sig 71 INFO: Key report: 3045022100d7fa049ecb98a195e8b4b5c21ff70ff644538ea38432802f032e321276767cf7022003b17cefd0d72413398761bff6969b43fe64bf2b73750e1d95c83581779850c0 INFO: Key report standard Pub size: 33, sig 72 INFO: Key report: 3046022100ff25141c631115b82ae54d7b9e0ec9ba36fc40450263b28db79505789b1395f5022100f6bf127e2c5237658fd58e54885e45650c48c51e2807fd1b2a28d65e91711a7f INFO: Key report standard Pub size: 33, sig 71 INFO: Key report: 3045022037139d573044b5d6da4c8893103c21f3f4c6f0edc27d10d6565b34fa37925ea00221008b16111cba474391cbf8ccd842cafeae00647e2e2503b9e84a93f5a4ec78ddaa INFO: Key report standard Pub size: 33, sig 70 INFO: Key report: 304402204d5338e43ca0cf40f80f49fc39406f850c856614dfeda92d44e7117639041725022027d10b6fe01bc6ef74fc9a42bad9f308858313cbce1d7c2a124d854b72f7c2de INFO: Key report standard Pub size: 33, sig 72 INFO: Key report: 3046022100ca1ce9f11ac78913816de520d8503d2336f5a4ef93ff5118f935e21baba396e6022100a3bc326b36351898978b59c2f6cdd2bc1381bdb82ef5d6a06e66f918952a3c61 INFO: Key report standard Pub size: 33, sig 70 INFO: Key report: 30440220333462295444174b6b3c630f004d1dfad00f86a14226f2901b9822e69e30c38e02202534ca04184b7cee6b3245ddba8ccaf9ae7858e78a24c5c6184c75a96316cbc0```
It isn't consistent enough for me to just smash bytes together without understanding. :wink:
well...not helpful and confusing: ```INFO: Key report: 304502206fc9f218afe66dd0e5087a5fbf6a00143b4a106a5c5b826f805cf4ecfc444dd7022100d6a4b065f0992f201e6553f3e68fa32e901c7f4ff63f571f080f2fcbe99be815 Oct 22, 2020 7:40:47 PM lib.test.KeyUtilTest testKeyPair INFO: Key report: ASN1StreamParser class org.bouncycastle.asn1.DLSequenceParser class org.bouncycastle.asn1.ASN1Integer Value: 50563533340942111709280476766673954330170109732405286366107419079320090725847 class org.bouncycastle.asn1.ASN1Integer Value: 97085929971092276940856138640916961836293272865594586885842012481817623128085```
i think i'm actually wrong about the packaging stuff
i think i now got around to where you were earlier about the swagger not fulfilling the requirements of the glassfish container in regards to what it needs to implement for that interface - i guess it's throwing a 500 while trying to serve a 404
so maybe it's missing an implicit bit of glassfish or something? i'll start taking blind stabs at that it's been a while since i've tried to figure managed java stuff out, and even then i was mostly doing jruby
yeah, I've avoided most of this application server j2ee ish stuff
jetty used to be its own container way back when, did glassfish get absorbed upstream into oracle at some point?
I have no idea
with the glassfish angle i'm starting to get more and more things saying this is a pretty common hurdle to hit and i think it's pretty much a matter of trying to jam the correct bits in
current best guess is that there's no default 404 provider and nothing implements a 404
or no json transform chain, that'll be the next i try
any other endpoint i can tickle to verify things other than 404 work?
so, what happened there is that swagger indeed requires something to implement the content transforms and nothing got pulled in which implemented the json transform for the content output
i guess that's some curried methods stuff where it is able to inject that dependency in out of thin air at runtime
Yeah, see curl/network.list
In the got repo
i'm doing a non-cached full rebuild of the container just to make sure
is that 405 from you when one tries that with the incorrect content negotiation? :smile:
i guess that's sorta correct the right response?
Not my doing. I don't care what people want, they get what they get
So the curl scripts I put in there set the headers
network/list returns a valid json as well
i already started to try to break the content negotiation as the root gives a json 404 for a text/http request :stuck_out_tongue:
is network/list a good enough smoketest to call it working?
Turns out `io.swagger` was not able to render JSONs as there was no rendering provider bundled in. Adding jersey-media-json-jackson 2.32 as a runtime dependency to the Maven POM fixes the .war and the container from https://github.com/snowblossomcoin/rosesnow/pull/1 is able to serve the content now.
if you're happy with the container declaration, you might as well merge both
did you jam the dockerhub credentials into the secrets management thingy on github already?
yeah, network/list is a good test
[snowblossomcoin/rosesnow] #2 Add jersey-media-json-jackson 2.32 to the Maven POM as a runtime dep
*https://github.com/snowblossomcoin/rosesnow/compare/3dc9f479ebae...8c2dc9ef304a* https://github.com/snowblossomcoin/rosesnow/commit/6a6adeacbdc3bfe07482b6eabcec3b7acdc263f5 - Use ARG for build time environment variables in the Dockerfile. https://github.com/snowblossomcoin/rosesnow/commit/a03611049f2f44e99220cb8dbb87d820eef071b9 - Silence an apt-key security warning in the Dockerfile. https://github.com/snowblossomcoin/rosesnow/commit/a1903f098c938125ee763a22939b103ff21d32f0 - Silence curl output and follow redirections in the Dockerfile. https://github.com/snowblossomcoin/rosesnow/commit/b97ea00a2a26e9604781586425bcd465394c4723 - Move to using apt-get -qq in the Dockerfile. https://github.com/snowblossomcoin/rosesnow/commit/acb75166490d857cb1bb46e7818d4dbc47dc701c - Install everything as late as possible in the Dockerfile. https://github.com/snowblossomcoin/rosesnow/commit/ec8e46d349f6632689ef97033c19933dcc08c8cc - Linebreak the installed packages in the Dockerfile for clarity. https://github.com/snowblossomcoin/rosesnow/commit/ae81801731f88f33b4de540edc99372bf4d88191 - Chain apt-get update and install in the Dockerfile to save a layer. https://github.com/snowblossomcoin/rosesnow/commit/7ac9d4f9c11fabd83eb9d0acb534a8cc87bb5770 - Avoid apt-get installing recommends and suggests in the Dockerfile. https://github.com/snowblossomcoin/rosesnow/commit/a3c4124529a530f73b02a734d32eef2a887827a2 - Use a headless JRE in the Dockerfile. https://github.com/snowblossomcoin/rosesnow/commit/61b2b11399e7ab3edb550b445ab8cd32677156b4 - Clean up transient apt-get metadata in the Dockerfile. https://github.com/snowblossomcoin/rosesnow/commit/784cd7c6e2f9ffb9444f957bfcbd368a3ba18b82 - Improve Dockerfile readability with linebreaks. https://github.com/snowblossomcoin/rosesnow/commit/16656f8c216e93c04fbb49b0cdc13be6401a0e6e - Add a non-privileged build user to the Dockerfile. https://github.com/snowblossomcoin/rosesnow/commit/db92946a7c5e859142e3f2832529069574818e6e - Parameterize and optimize the git clone. https://github.com/snowblossomcoin/rosesnow/commit/bf07c9c64a9c923fbea31cfffab6ddf9304e7e23 - Use the Jetty 9 from Ubuntu 20.04 as the runtime in the Dockerfile. https://github.com/snowblossomcoin/rosesnow/commit/8c2dc9ef304a723643f18c3d2025f7c4b66a344b - Merge pull request #1 from Rotonen/2020-10-20-improve-dockerfile
[snowblossomcoin/rosesnow] #1 Improve Dockerfile
awesome
once you do this long enough, you start to duck type across ecosystems
like, i have ~zero idea of java or maven, but the shape of the problem was familiar
yeah, I used maven a little bit a few years ago but never liked it
felt like I needed a 200 line xml config to do normal things like run tests and build things
so "latest" is the implicit tag name if you don't specify?
With your change, it looks like /network/list is showing some sort of API information. It should return: ```{ "network_identifiers": [ { "blockchain": "snowblossom", "network": "mainnet" }, { "blockchain": "snowblossom", "network": "testnet" } ] }```
well, that's certainly not expected then
looked legit as it returned hashes etc.
well, more things to jam in there, i suppose - the json renderer is clearly a step forward
agreed
i don't have the container up, can you paste the formatted output?
i didn't bother piping that to jq as i was on windows
a clue: ```21:00:45.477 [main] INFO i.s.o.i.c.OpenAPIOperationController - looking for method: `public ResponseContext constructionHash(io.swagger.oas.inflector.models.RequestContext request, com.fasterxml.jackson.databind.JsonNode [simple type, class com.fasterxml.jackson.databind.JsonNode])` in class `io.swagger.oas.sample.controllers.Default` 21:00:45.477 [main] WARN i.s.o.i.utils.ReflectionUtils - Failed to find class [io.swagger.oas.sample.controllers.Construction] 21:00:45.478 [main] WARN i.s.o.i.utils.ReflectionUtils - Failed to find class [io.swagger.oas.sample.controllers.ConstructionController] 21:00:45.478 [main] WARN i.s.o.i.utils.ReflectionUtils - Failed to find class [default] 21:00:45.478 [main] WARN i.s.o.i.utils.ReflectionUtils - Failed to find class [default] 21:00:45.478 [main] INFO i.s.o.i.c.OpenAPIOperationController - looking for method: `public ResponseContext constructionSubmit(io.swagger.oas.inflector.models.RequestContext request, com.fasterxml.jackson.databind.JsonNode [simple type, class com.fasterxml.jackson.databind.JsonNode])` in class `io.swagger.oas.sample.controllers.Default` 21:00:45.479 [main] WARN i.s.o.i.utils.ReflectionUtils - Failed to find class [io.swagger.oas.sample.controllers.Call] 21:00:45.480 [main] WARN i.s.o.i.utils.ReflectionUtils - Failed to find class [io.swagger.oas.sample.controllers.CallController] 21:00:45.480 [main] WARN i.s.o.i.utils.ReflectionUtils - Failed to find class [default] 21:00:45.480 [main] WARN i.s.o.i.utils.ReflectionUtils - Failed to find class [default]```
Looks like it isn't finding the controllers
i meant the weird json output you got, or is it just the stack trace?
this is a console log
I'll get the json
so something needs to also provide a http responder or something like that
which is sorta weird, as it does respond, but sure
I got these errors before when I was having trouble getting the introspection to find the controllers and their classes
smells like dependency injection to me, so it just picks something which implements the correct interface
it should be looking in io.swagger.controllers.Call rather than io.swagger.oas.sample.controllers.Call
```{ "name": null, "attribute": false, "wrapped": false, "wrappedName": null, "typeName": "object", "values": { "network_identifiers": { "name": null, "attribute": false, "wrapped": false, "wrappedName": null, "typeName": "array", "items": [ { "name": null, "attribute": false, "wrapped": false, "wrappedName": null, "typeName": "object", "values": { "blockchain": { "name": null, "attribute": false, "wrapped": false, "wrappedName": null, "typeName": "string", "value": "bitcoin", "namespace": null, "prefix": null }, "sub_network_identifier": { "name": null, "attribute": false, "wrapped": false, "wrappedName": null, "typeName": "object", "values": { "metadata": { "name": null,```
That is the first 40 lines anyways. It looks like a description of the openapi call
i just went with "says blockchain is probably fine" :smile:
ah, it looks like inflector.yaml isn't making it into the war
that would even make sense
i guess one would also need to put the xml and yaml entity processors as runtime deps?
we shall see. The inflection service is clearly starting and looking for things. I imagine it has what it needs.
now you have two inflector.yamls, is that ok? or could the old one be removed?
mostly to avoid ever editing the wrong one
maybe, I don't want to break local run
symlink
```1:29:22.154 [main] WARN io.swagger.v3.parser.OpenAPIV3Parser - Exception while reading: java.lang.RuntimeException: Could not find ./src/main/resources/openapi.yaml on the classpath```
i guess you just spotted that yourself too
i think i now again get the rosetta defaults of something ```λ curl -d "{}" -H "Content-Type: application/json" http://localhost:8080/network/list/ {"network_identifiers":[{"blockchain":"bitcoin","sub_network_identifier":{"metadata":{"producer":"0x52bc44d5378309ee2abf1539bf71de1b7d7be3b5"},"network":"shard 1"},"network":"mainnet"},{"blockchain":"bitcoin","sub_network_identifier":{"metadata":{"producer":"0x52bc44d5378309ee2abf1539bf71de1b7d7be3b5"},"network":"shard 1"},"network":"mainnet"}]}```
weird ass thing. Now I get the bitcoin sample text.
yeah, clearly so do you
and the root 404 is html again
Failed to find class [io.swagger.controllers.Call]
so content negotiation now works
```looking for method: `public ResponseContext mempoolTransaction(io.swagger.oas.inflector.models.RequestContext request, com.fasterxml.jackson.databind.JsonNode [simple type, class com.fasterxml.jackson.databind.JsonNode])` in class `io.swagger.controllers.Default``` i think the `in class io.swagger.controllers.Default` in those is the key thing
```loading inflector config from inflector.yaml 21:40:08.544 [main] WARN i.s.o.inflector.config.Configuration - couldn't read inflector config from system property 21:40:08.601 [main] ERROR i.s.o.inflector.config.Configuration - unable to add mapping for `DefinitionFromSwaggerSpecification` : `fully.qualified.path.to.Model`, fully.qualified.path.to.Model```
so swagger config issue?
or broken inflector yaml?
probably not bringing in the bazel deploy jar
yep
starts to sound like the final hurdle
yeah
why'd you jam the `echo cor` in there? :smile:
I was having trouble avoiding the docker caching that step
so changing the command makes it redo it
ha
that's exactly what i changed the clone to an init -> fetch -> checkout for, pass the git commit hash as REFSPEC to the build args
yeah
or if you're patient, add `--no-cache` to the build
I'm not. But the bazel from scratch is the slowest part so it doesn't really matter.
bazel is very fast on repeat runs but a little slow on the first pass
well, the apt-get installs hit the network for a bit too
java + bazel + maven is big
as long as it works, I don't think anyone will care
the runtime container is significantly less heavy as it only has jetty and that war in it
yeah, and it starts up fast
Looks like user can't create /data
correct, arbitrary users cannot write to root on ubuntu
the jetty user owns all of `/var/lib/jetty9/`
maybe make the db directory configurable through environment variables and ENV it to `/var/lib/jetty9/rosesnow/db` on the runtime? the war is under `/var/lib/jetty9/webapps/ROOT.war`