Strange bottleneck copying files on EFS shares
A bit of a weird problem that has popped up out of nowhere and will little to no information. I work in support of a PHP (Laravel) application that stores asset files for our users. The application runs on across a few EC2 instances, and the assets are stored in EFS mounts.
The other day, one of our core routes which is responsible for triggering the server to copy a directory from A to B began to take a significant amount of time (from 5s to 90s) with no changes to the contents or application code.
I was able to isolate the delay to an xcopy call that moves 1 folder on the EFS to another folder in the same EFS. When I reached out to Amazon for assistance looking at the EFS metrics in CloudWatch, they indicated that everything looked strong, but they noted the fall off. Throughput is provisioned at 10Mb/s, and I've only spiked at 4-5Mb/s in the past few days.
Using NetData, I'm able to observe performance of RAM, CPU, NFS Client calls, and much more. From what I can tell, the NFS client is taking its sweet time. When testing locally without the EFS, the call is very speedy. I tried to modify the app code so that rather than a PHP copy, it would perform a system call to cp. This didn't seem to work.
Has anyone else experienced random latency issues like this before when using EFS? Does anyone have any recommendations on debugging techniques to see why this might be happening?
Thank you for reading and information you can offer up.
php laravel filesystems latency amazon-efs
asked Nov 20 at 21:23
Neurax
6471516
add a comment |
A bit of a weird problem that has popped up out of nowhere and will little to no information. I work in support of a PHP (Laravel) application that stores asset files for our users. The application runs on across a few EC2 instances, and the assets are stored in EFS mounts.
The other day, one of our core routes which is responsible for triggering the server to copy a directory from A to B began to take a significant amount of time (from 5s to 90s) with no changes to the contents or application code.
I was able to isolate the delay to an xcopy call that moves 1 folder on the EFS to another folder in the same EFS. When I reached out to Amazon for assistance looking at the EFS metrics in CloudWatch, they indicated that everything looked strong, but they noted the fall off. Throughput is provisioned at 10Mb/s, and I've only spiked at 4-5Mb/s in the past few days.
Using NetData, I'm able to observe performance of RAM, CPU, NFS Client calls, and much more. From what I can tell, the NFS client is taking its sweet time. When testing locally without the EFS, the call is very speedy. I tried to modify the app code so that rather than a PHP copy, it would perform a system call to cp. This didn't seem to work.
Has anyone else experienced random latency issues like this before when using EFS? Does anyone have any recommendations on debugging techniques to see why this might be happening?
Thank you for reading and information you can offer up.
php laravel filesystems latency amazon-efs
asked Nov 20 at 21:23
Neurax
6471516
add a comment |
A bit of a weird problem that has popped up out of nowhere and will little to no information. I work in support of a PHP (Laravel) application that stores asset files for our users. The application runs on across a few EC2 instances, and the assets are stored in EFS mounts.
The other day, one of our core routes which is responsible for triggering the server to copy a directory from A to B began to take a significant amount of time (from 5s to 90s) with no changes to the contents or application code.
I was able to isolate the delay to an xcopy call that moves 1 folder on the EFS to another folder in the same EFS. When I reached out to Amazon for assistance looking at the EFS metrics in CloudWatch, they indicated that everything looked strong, but they noted the fall off. Throughput is provisioned at 10Mb/s, and I've only spiked at 4-5Mb/s in the past few days.
Using NetData, I'm able to observe performance of RAM, CPU, NFS Client calls, and much more. From what I can tell, the NFS client is taking its sweet time. When testing locally without the EFS, the call is very speedy. I tried to modify the app code so that rather than a PHP copy, it would perform a system call to cp. This didn't seem to work.
Has anyone else experienced random latency issues like this before when using EFS? Does anyone have any recommendations on debugging techniques to see why this might be happening?
Thank you for reading and information you can offer up.
php laravel filesystems latency amazon-efs
asked Nov 20 at 21:23
Neurax
6471516
A bit of a weird problem that has popped up out of nowhere and will little to no information. I work in support of a PHP (Laravel) application that stores asset files for our users. The application runs on across a few EC2 instances, and the assets are stored in EFS mounts.
The other day, one of our core routes which is responsible for triggering the server to copy a directory from A to B began to take a significant amount of time (from 5s to 90s) with no changes to the contents or application code.
I was able to isolate the delay to an xcopy call that moves 1 folder on the EFS to another folder in the same EFS. When I reached out to Amazon for assistance looking at the EFS metrics in CloudWatch, they indicated that everything looked strong, but they noted the fall off. Throughput is provisioned at 10Mb/s, and I've only spiked at 4-5Mb/s in the past few days.
Using NetData, I'm able to observe performance of RAM, CPU, NFS Client calls, and much more. From what I can tell, the NFS client is taking its sweet time. When testing locally without the EFS, the call is very speedy. I tried to modify the app code so that rather than a PHP copy, it would perform a system call to cp. This didn't seem to work.
Has anyone else experienced random latency issues like this before when using EFS? Does anyone have any recommendations on debugging techniques to see why this might be happening?
Thank you for reading and information you can offer up.
php laravel filesystems latency amazon-efs
php laravel filesystems latency amazon-efs
asked Nov 20 at 21:23
Neurax
6471516
asked Nov 20 at 21:23
Neurax
6471516
asked Nov 20 at 21:23
Neurax
6471516
asked Nov 20 at 21:23
Neurax
6471516
asked Nov 20 at 21:23
Neurax
6471516
6471516
add a comment |
add a comment |
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401750%2fstrange-bottleneck-copying-files-on-efs-shares%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401750%2fstrange-bottleneck-copying-files-on-efs-shares%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
