Day 68 of DevOps, Day 18 of Azure. Kubernetes closed yesterday at Day 67, and Jenkins opens here and runs to Day 81. Both of today's tasks ended the same way: the tool said it had succeeded, and the check that settled it was something else entirely.
One Jenkins task, one Azure task. Install Jenkins and get through the setup wizard, then copy a single file into a blob container. The tasks come from the KodeKloud Engineer platform.
Task 1: Install Jenkins
The task: install Jenkins on the jenkins server, start it with the service command, and create a named first administrator.
Step 1: Java, because nothing else works without it
ssh root@jenkins
java -version
-bash: java: command not found
sudo apt update
sudo apt install fontconfig openjdk-21-jre
java -version
openjdk version "21.0.10" 2026-01-20
OpenJDK Runtime Environment (build 21.0.10+7-Ubuntu-124.04)
The installation page lists "Java 21 or later" and gives that exact install line, fontconfig openjdk-21-jre.
The version is worth a moment rather than a copy and paste. Jenkins' Java support policy lists Java 17 under unsupported versions, and names its final LTS release as 2.541.3. That is the version this lab installed. So this server is sitting on the last LTS that would still have accepted Java 17, and from the next one Java 21 is not a recommendation, it is the floor.
Step 2: The repository and the package
sudo wget -O /etc/apt/keyrings/jenkins-keyring.asc \
https://pkg.jenkins.io/debian-stable/jenkins.io-2026.key
echo "deb [signed-by=/etc/apt/keyrings/jenkins-keyring.asc]" \
https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
/etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt update
sudo apt install jenkins
Get:1 https://pkg.jenkins.io/debian-stable binary/ jenkins 2.541.3 [95.8 MB]
Two things in those four commands are easy to get wrong from an old tutorial. debian-stable is the Jenkins LTS apt repository, not anything belonging to Debian or Ubuntu, so none of this comes from the distribution archives. And the key filename carries a year, jenkins.io-2026.key, which is exactly the sort of value a copied script keeps long after it has moved on.
Step 3: Read the install output before scrolling past it
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Created symlink /etc/systemd/system/multi-user.target.wants/jenkins.service -> /usr/lib/systemd/system/jenkins.service
Those first two lines say the package tried to start the service and the environment refused. That is a container convention rather than a failure, and it is why the next command reported Jenkins not running immediately after a clean install.
service jenkins status
* jenkins is not running
service jenkins start
service jenkins status
* Starting Jenkins Automation Server jenkins Setting up max open files limit to 8192 [ OK ]
* jenkins is running
The task asked for service. The documentation does not use it anywhere, and says why: "Beginning with Jenkins 2.335 and Jenkins 2.332.1, the package is configured with systemd rather than the older System V init." service jenkins status works through systemd's compatibility shim. systemctl status jenkins is what the docs would have me type.
Step 4: Unlock, and notice where the password lives
cat /var/lib/jenkins/secrets/initialAdminPassword
Jenkins listens on 8080 and refuses to do anything until that value is pasted into the browser. The design is deliberate: the only person who can finish the setup is someone who can already read the server's filesystem. Which is also the reason not to leave a half-configured Jenkins reachable while you go and find the file.
Then "Install suggested plugins", described in the docs as "the recommended set of plugins, which are based on most common use cases", and the first administrator user. Day 69 is about what happens when the suggested set does not contain what you need.
What the docs recommend
Use systemctl. The package has been a systemd unit since 2.332.1, and systemctl cat jenkins shows you the unit the package installed, which is where the port and the Java options actually live.
Know where JENKINS_HOME is before you need it. "On Ubuntu by default, this is set to /var/lib/jenkins", and it is "the root of the disk directory structure that Jenkins uses to perform builds and store archives". Everything that matters is in there: jobs, credentials, plugins, the lot. It is the backup target, and on Day 71 it is also where an SSH key generated by the jenkins user ends up.
Treat the suggested plugin set as a starting point rather than a decision. It is built for common cases, and the next two days are both about reaching past it.
Task 2: Copy a file into a blob container
The task: copy /tmp/xfusion.txt into the existing container xfusion-blob-398826849.
Step 1: Upload
az storage blob upload --account-name $SA --account-key $KEY \
--container-name $CONTAINER --name xfusion.txt --file /tmp/xfusion.txt
Step 2: Do not stop at the listing
The listing showed the right name and 33 bytes, matching the local file. That is good evidence and it is not proof. A file can be the right length with different bytes in it, and text tools hide trailing whitespace, line-ending changes and encoding shifts.
az storage blob download --account-name $SA --account-key $KEY \
--container-name $CONTAINER --name xfusion.txt --file /tmp/roundtrip.txt
cmp -s /tmp/xfusion.txt /tmp/roundtrip.txt && echo "MATCH: byte-identical"
MATCH: byte-identical
cmp compares bytes, which is the only check that means anything when the claim is that nothing changed. Cheap here at 33 bytes. For a large upload the equivalent is comparing checksums rather than re-downloading.
The checksum that was not where I looked
--query "{ETag:etag,Md5:contentMd5}"
{ "ETag": "\"0x8DF1C7E15C4FC83\"", "Md5": null }
So I read it off the blob instead, which worked:
az storage blob show ... --query "properties.contentSettings.contentMd5" -o tsv
My conclusion at the time was that contentMd5 is not in the upload response. Checking it, that is wrong. The Put Blob REST reference lists Content-MD5 among its response headers, "Returned for a block blob so that the client can check the integrity of the message content", and adds that "In version 2012-02-12 and later, this header is returned even when the request doesn't include Content-MD5 or x-ms-blob-content-md5 headers".
So the API did return it, and the CLI did not surface it under the name I queried. That makes the finding sharper rather than weaker: what a command prints is the command's own shape, and it is not a reliable guide to what the service sent back. The habit that follows is unchanged, and now rests on better ground:
az <command> -o json | grep -i <fragment>
Hex and base64 are the same digest
md5sum /tmp/xfusion.txt
2eeef38a56ad6c682ecd2cf611c9f921
md5sum prints hex. Azure wants Base64, and the requirement is explicit in the error table: InvalidMd5 means "The MD5 value specified in the request is invalid. The MD5 value must be 128 bits and Base64-encoded." The neighbouring error, Md5Mismatch, is what a well-formed but wrong digest gets.
openssl md5 -binary /tmp/xfusion.txt | base64
Worth knowing before deciding that a checksum mismatch means corruption.
What the docs recommend
Use conditional headers rather than an overwrite flag. The ETag is the concurrency token: "Each version of the blob has a unique tag, called an ETag, that you can use with access conditions to assure you only change a specific instance of the blob."
--if-match '"0x8DF1C7E15C4FC83"' # only write if unchanged since I read it
--if-none-match '*' # only write if the blob does not exist
The wildcard is documented directly: "Specify the wildcard character (*) to perform the operation only if the resource doesn't exist, and fail the operation if it does exist." An unmet condition returns 412.
That beats relying on --overwrite, whose default is not uniform. On az storage blob upload it defaults to false and "the operation will fail with ResourceExistsError"; on az storage blob download the docs give the default as true. Per command, again.
Pick the blob type deliberately, because you only get one chance. "You specify the blob type when you create the blob. Once the blob has been created, its type cannot be changed." Block blobs for files, append blobs for logs, page blobs for virtual machine disks. A log sink created as a block blob cannot later be appended to efficiently.
Remember the namespace is flat. "By default, the Blob service is based on a flat storage scheme, not a hierarchical scheme. However, you may specify a character or string delimiter within a blob name to create a virtual hierarchy." A slash in a blob name is a character. There is no empty folder, deleting the last blob under a prefix makes the folder disappear, and renaming a folder means copying everything under it.
If you need real directories, the upgrade path now exists, which corrects something I had written down as fixed. Hierarchical namespace used to be a creation-time choice only. Microsoft now documents az storage account hns-migration start --type upgrade, with the warning that "An upgrade is one-way. There's no way to revert your account once you've performed the upgrade", and preconditions: no page blobs in the account, and snapshots, encryption scopes, immutable storage and soft delete disabled first.
And know where this command stops. For a directory there is az storage blob upload-batch. For anything large, there is AzCopy, where resume is documented: azcopy jobs resume "Resumes the existing job with the given job ID", and on resume it "attempts to transfer all of the files that are listed in the plan file and weren't already transferred".
What I learned
Both halves handed me a success message I could not use. service jenkins status said it was running, which is true and says nothing about whether the setup wizard will load. The blob listing said 33 bytes, which is true and says nothing about the bytes.
The second one also cost me a conclusion. I had written down that the upload response does not carry the checksum, when what actually happened is that the response carried it and the CLI printed something else. Those look identical in the terminal, and they are not the same fact.
So here is the Day 68 question. When a command prints a field as null, do you check whether the service sent it, or do you just look somewhere else and move on?
Day 68 down. Thirty-two to go.
