Currently ROR download links are provided only on request, after submission via ROR website. Most of them will be hosted on AWS and the links expire after some time. Why not provide direct download links with a)latest version for specific ES version b)specific ROR version for specific ES version? I know that Simone does not recommend using outdated versions, but if I have done my testing in DEV region with some specific version, I would definitely want to install the same version in my SIT/UAT/PROD, rather than just pulling the latest version everytime.
Given how frequent the fixes come in and this change has to be deployed in multiple servers, I am planning to build some scripts that can automate the part of downloading the latest version, remove old version and install the new version of ROR. This will be partially useful during first time install as most of the work will be on setting up ROR config files. But subsequent updates, in majority of the cases, will just involve updating ROR version with no changes to config files. So I feel that having these fixed download links is going to be useful. The way that I plan to do it is look for files locally first. If not available, then download.
BTW, just FYI, these kind of direct download links are available for both XPack and Searchguard. XPack provides it on their website, where as SG has it hosted on maven repo.
Hey @askids, how is relying on third party links in your automated deployment a good strategy?
I see this very brittle from a reliability standpoint: I might decide to move old builds to another folder in my web space, plus: do you want the reliability of ROR servers to be a variable in the repeatability/immutability of your automation?.
But also from a security standpoint: security software download sites are a very hot target for malware operators.
If I were you, like: all of you, I’d host the tested plugin version in a S3/web space I manage. Or much, much better: create an immutable docker image with the tested plugin, add it to a private docker registry and sleep better nights.
BTW, the best approach I saw was at CERN: they build the plugin from sources (checking out GIT tags) as part of the their deployment strategy. Am I right @psaiz, @schwicke?
@sscarduzio when you say 3rd party link, hope that you realize that you are talking about ROR only So if we cannot even rely on a secure link from ROR (be it hosted directly on ROR website or AWS link), how can we rely on the ROR plugin itself for security…JK
I am not saying that the script will download in all lanes. In our environment, we don’t even allow internet connectivity from dev servers, forget prod. These kind of upgrades are typically tested starting with a developer machine. So this is where the download option will be very useful and will be in play. Subsequently for the downloaded version, the script will utilize the local available zip file only for that version as that is what gets promoted. So in typical scenario, this upgrade then move on to a proper development server, SIT lane, UAT lane and finally production. So whatever was downloaded and tested in the 1st box is what gets eventually packaged and deployed into higher lanes.
We have currently automated the part of uninstall and installing using the local zip file and can upgrade to any version. But given that we went through so many upgrades starting from 1.16.9 to eventually 1.16.14, with multiple pre-release versions being tested in between, for me it makes sense to have that automated download option that developers could leverage to perform the upgrade on their individual machine.
Also, FYI, xpack plugin installation that is integrated with the windows MSI also downloads directly from Elastic site. I have open feature request with them to provide offline plugin installation capability because that is what we will use in proper server machines and that will simply the upgrade task on actual servers. But that download and install plugin option is still useful, when we have to upgrade and validate the new version on individual developer machines.
lol you have your point, I was just trying to put myself in the shoes of a very paranoid/principled dev.
Did I understand well what you are proposing? The user experience proposed would be:
Download ES version X.Y.Z
bin/elasticearch-plugin install readonlyrest
the plugin tool automatically pulls the latest plugin code for the ES version X.Y.Z
plugin installed, ready to run.
If this is true, as a user, you will have zero control in what ROR version you will be pulling, right? What if you tested 1.16.13 and last night I just released 1.16.14 and all of a sudden you’re promoting code you didn’t test?
The way that I am building it, I pass the version number as input to my script. So in that version input, I want to provide 2 variants. First one is simple get latest ROR version for ESx.y.z. Second option is more specific ROR 1.2.34 for ESx.y.z. Currently, I am only supporting offline install. So first check, if ROR is installed. If yes, remove it. Then install the new version and restart ES service.
In case of xpack, the msi is specific to a ES version. So the xpack that they download is also specific to that ES version.
With only download option available for xpack plugin install, they also have the same problem like you have described. But on their end, they don’t have separate xpack version just with a security fix. When they release it, they will release all products with the new version.
Does that mean that if a severe security bug gets discovered in x-pack for ES 5.6.4, there’s no way for them to deliver a fix to a 5.6.4 customer other than forcing them to update the whole Elastic stack?
Have you seen how slow are people upgrading their ES clusters? We have customers in 5.1.x, 5.3.x, and refused to deal with a dozen others on 2.x. And these guys do not plan to update their cluster anytime soon. Which I understand, because it’s a extremely difficult to upgrade an ES cluster without downtime.
I really think binding x-pack version tightly to each Elastic Stack release is a bad idea. ReadonlyREST won’t follow this path and all our customers using supported ES versions will be able to receive the latest plugin code.
Another reason we can’t follow the Elastic Stack versioning alone, is that we simply don’t have control on when/how often they’ll release.
About the frequent releases of ROR
I believe ROR is releasing coming out often is symptomatic that we’re working hard on the product. The more customers we onboard, the faster we go due to the great input from very knowledgeable customers like yourself.
Having said that, I recognise that upgrading to a new release implies engineering time on customer side, so we’ll will try to keep it down to max 1 per month. Unless major things need to be fixed.
At least that i what I have understood. If somebody else is currently using or previously used xpack for security, they should be able to confirm or correct my understanding.
Also, just wanted to add that it looks like there is some level of tolerance on version mismatch between the products. Some mismatch might just throw a warning and not actually stop the functionality. So technically, its possible for xpack to release a newer version and not needing to upgrade the entire Elastic stack. Again, our focus shouldn’t be on what xpack is doing as it contains many plugins packaged together and not just security. So they might have their own reasons for wanting customer to upgrade and not just related to security. Lets leave it at that as there is no one here to support their viewpoint.
Coming back to the original suggestion, Building from ROR code might not be a feasible option for everyone. Take us for example. We are on Windows Stack, running applications on .Net. We wouldn’t want to be dealing with intricacies of Java, if the build fails or even to set it up for that matter. Its not that it cannot be learnt, but its an unnecessary ask to be able to use ROR.
I would still recommend that you think about providing the direct download link options - one always for latest ROR (for ES X.Y.Z) and second being version specific (ROR 1.2.3.4 for ES X.Y.Z). For some versions, which you want to stop supporting OR has known critical vulnerabilities and want to force users to upgrade to a newer version, you can remove the links.
I don’t mind frequent updates as long as I can automate some or most part of the install process. This might not look very useful during first time setup. But all subsequent updates can be possibly made simple by providing this option. This could also ensure that the ROR customers can think about making the required security upgrade more frequently, if the process is less painful. This way, they are not stuck in some old version with known security vulnerabilities.
Sorry for the late reply. Indeed, we’ve decided to package not only readonlyRest but also all other plugins we use for ES and kibana as rpms. We build them from source. This allows us to control the version across all clusters and on all cluster nodes (and we have many of them).
@coutoPL looks like ROR Kibana Free edition, still asks for email id to be passed on as mandatory parameter, where as ROR ES free edition does not ask for it. Do you plan to retain the email check or make it consistent with what you have for ES edition?
This functionality seems to be broken. I am trying to download older version of the plugin. But its always downloading 1.23.0. Can you please check? This is working fine for Kibana plugin.
Thanks @sscarduzio. I was trying to troubleshoot the other issue on index patterns. So looks like it may not be related to ROR ES, but actually related to ROR Kibana. Since direct download link working fine for ROR Kibana, i can work with it. Thanks for getting back!
Now, the ROR_Kbn link seems to be broken. Getting internal server error.
Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
So if we cannot even rely on a secure link from ROR (be it hosted directly on ROR website or AWS link), how can we rely on the ROR plugin itself for security…JK